{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 🎬 AI-Powered Real-time Video Stream Intelligence & Incident Detection System\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/agents/video_intelligence_agent.ipynb)\n",
        "\n",
        "[![AI Learning Hub For Developers](https://img.shields.io/badge/AI%20Learning%20Hub%20For%20Developers-Click%20Here-blue)](https://www.mongodb.com/resources/use-cases/artificial-intelligence?utm_campaign=ai_learning_hub&utm_source=github&utm_medium=referral)\n",
        "\n",
        "An intelligent video monitoring system that provides real-time analysis and incident detection for live video streams, broadcasts, and recordings. The system combines  AI embeddings, and languge models to automatically detect, analyze, and resolve video quality issues, network problems, and streaming incidents as they occur.\n",
        "\n",
        "This notebook implements an **AI-Powered Real-time Video Stream Intelligence & Incident Detection System** that automatically monitors and analyzes video content to detect incidents, quality issues, and network problems. The system extracts frames from videos at regular intervals, generates semantic embeddings using Voyage AI's multimodal models, and creates detailed scene descriptions using OpenAI's GPT-4 Vision. \n",
        "\n",
        "- **Users can search through video content using natural language queries like \"find frames with a referee\" and instantly jump to relevant timestamps through an interactive HTML5 video player that displays similarity scores and scene descriptions.**\n",
        "\n",
        "The system supports multiple video sources including local files, webcams, and YouTube livestreams, storing all analysis data in MongoDB with vector search capabilities for fast retrieval. Built using an agent-based architecture, specialized AI agents handle different aspects like frame retrieval, video display, incident analysis, and stream monitoring. \n",
        "- **Real-time monitoring continuously processes live video feeds, compares frames against a database of known incidents, and provides immediate alerts with visual overlays.**\n",
        "\n",
        "This makes the system valuable for broadcast monitoring, security surveillance, quality assurance, and content discovery applications where organizations need to automatically detect issues, search large video archives, or monitor live streams for technical problems or security incidents.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Pre-requisite:\n",
        "- Please ensure that you are using a MongoDB 8.1 database to use the new $rankFusion operator"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "! pip install -Uq pymongo voyageai pandas datasets opencv-python pillow openai openai-agents yt-dlp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "import getpass\n",
        "import os\n",
        "\n",
        "\n",
        "# Function to securely get and set environment variables\n",
        "def set_env_securely(var_name, prompt):\n",
        "    value = getpass.getpass(prompt)\n",
        "    os.environ[var_name] = value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 1: Extracting Embeddings and Metadata from Video Data\n",
        "\n",
        "This step involves extracting video frames and generating corresponding embeddings and metadata descriptions to facilitate intelligent search functionality.\n",
        "\n",
        "This step covers three key techniques:\n",
        "\n",
        "- Video-to-frame conversion for image extraction\n",
        "- Multimodal embedding generation with Voyage AI to encode semantic relationships between text and images\n",
        "- Automated metadata generation using GPT-4o Vision Pro\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 1.1 Video to Images Function Explanation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "This function: `video_to_images` extracts still images from a video at regular time intervals (default: every 2 seconds).\n",
        "\n",
        "What it does:\n",
        "\n",
        "1. Opens a video file using OpenCV\n",
        "2. Calculates which frames to extract based on the video's frame rate and desired time interval\n",
        "3. Loops through the video, saving only the frames that match the timing interval\n",
        "4. Saves each extracted frame as a JPEG with a timestamp filename (e.g., \"frame_0001_t2.0s.jpg\")\n",
        "5. Returns the total number of frames extracted\n",
        "\n",
        "Key parameters:\n",
        "\n",
        "- `video_path`: Input video file\n",
        "- `output_dir`: Where to save the images (default: \"frames\")  \n",
        "- `interval_seconds`: Time between extractions (default: 2 seconds)\n",
        "\n",
        "Usage:\n",
        "\n",
        "The example usage extracts frames every 2 seconds from \"videos/video.mp4\" and saves them to a \"frames\" folder. This is useful for creating video thumbnails or analyzing video content frame by frame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from pathlib import Path\n",
        "\n",
        "import cv2\n",
        "\n",
        "\n",
        "def video_to_images(video_path, output_dir=\"frames\", interval_seconds=2):\n",
        "    \"\"\"\n",
        "    Convert a video to images by extracting frames every specified interval.\n",
        "\n",
        "    Args:\n",
        "        video_path (str): Path to the input video file\n",
        "        output_dir (str): Directory to save extracted frames (default: \"frames\")\n",
        "        interval_seconds (float): Time interval between frame extractions (default: 2 seconds)\n",
        "\n",
        "    Returns:\n",
        "        int: Number of frames extracted\n",
        "    \"\"\"\n",
        "\n",
        "    # Create output directory if it doesn't exist\n",
        "    Path(output_dir).mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "    # Open the video file\n",
        "    cap = cv2.VideoCapture(video_path)\n",
        "\n",
        "    if not cap.isOpened():\n",
        "        raise ValueError(f\"Error: Could not open video file {video_path}\")\n",
        "\n",
        "    # Get video properties\n",
        "    fps = cap.get(cv2.CAP_PROP_FPS)\n",
        "    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n",
        "    duration = total_frames / fps\n",
        "\n",
        "    print(f\"Video info: {fps:.2f} FPS, {total_frames} frames, {duration:.2f} seconds\")\n",
        "\n",
        "    # Calculate frame interval\n",
        "    frame_interval = int(fps * interval_seconds)\n",
        "\n",
        "    frame_count = 0\n",
        "    extracted_count = 0\n",
        "\n",
        "    try:\n",
        "        while True:\n",
        "            ret, frame = cap.read()\n",
        "\n",
        "            if not ret:\n",
        "                break\n",
        "\n",
        "            # Extract frame every interval_seconds\n",
        "            if frame_count % frame_interval == 0:\n",
        "                # Create filename with timestamp\n",
        "                timestamp = frame_count / fps\n",
        "                filename = f\"frame_{extracted_count:04d}_t{timestamp:.1f}s.jpg\"\n",
        "                filepath = os.path.join(output_dir, filename)\n",
        "\n",
        "                # Save frame as image\n",
        "                cv2.imwrite(filepath, frame)\n",
        "                extracted_count += 1\n",
        "                print(f\"Extracted frame {extracted_count}: {filename}\")\n",
        "\n",
        "            frame_count += 1\n",
        "\n",
        "    finally:\n",
        "        cap.release()\n",
        "\n",
        "    print(f\"Extraction complete! {extracted_count} frames saved to '{output_dir}'\")\n",
        "    return extracted_count"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the next cells we will be downloading a video to use for the video intelligence use case\n",
        "\n",
        "Video for this use case is obtained from YouTube, but you can modify the cells below for your own use case"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from pathlib import Path\n",
        "\n",
        "import yt_dlp\n",
        "\n",
        "\n",
        "def download_youtube_video(\n",
        "    url: str, output_dir: str = \"videos\", quality: str = \"best\"\n",
        ") -> dict:\n",
        "    \"\"\"\n",
        "    Download a YouTube video to `output_dir` at the given `quality`.\n",
        "    Falls back to 'best' if the requested format isn't available.\n",
        "    \"\"\"\n",
        "    Path(output_dir).mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "    def make_opts(fmt):\n",
        "        return {\n",
        "            \"format\": fmt,\n",
        "            \"outtmpl\": str(Path(output_dir) / \"%(title)s.%(ext)s\"),\n",
        "            \"noplaylist\": True,\n",
        "            **{\n",
        "                k: False\n",
        "                for k in (\n",
        "                    \"extractaudio\",\n",
        "                    \"writeinfojson\",\n",
        "                    \"writesubtitles\",\n",
        "                    \"writeautomaticsub\",\n",
        "                    \"ignoreerrors\",\n",
        "                )\n",
        "            },\n",
        "        }\n",
        "\n",
        "    # Determine initial format string\n",
        "    fmt = \"best[height<=1080]\" if quality == \"best\" else quality\n",
        "\n",
        "    try:\n",
        "        with yt_dlp.YoutubeDL(make_opts(fmt)) as ydl:\n",
        "            info = ydl.extract_info(url, download=False)\n",
        "            if not info:\n",
        "                return {\"success\": False, \"error\": \"No metadata extracted\", \"url\": url}\n",
        "            if info.get(\"is_live\"):\n",
        "                return {\n",
        "                    \"success\": False,\n",
        "                    \"error\": \"Live streams not supported\",\n",
        "                    \"title\": info.get(\"title\"),\n",
        "                    \"url\": url,\n",
        "                    \"is_live\": True,\n",
        "                }\n",
        "            # Warn on long videos\n",
        "            if info.get(\"duration\", 0) > 3600:\n",
        "                print(\"⚠️ Video >1h; this may take a while.\")\n",
        "\n",
        "            # Attempt download\n",
        "            print(f\"⬇️ Downloading at format `{fmt}`…\")\n",
        "            ydl.download([url])\n",
        "\n",
        "    except yt_dlp.utils.DownloadError as e:\n",
        "        err = str(e)\n",
        "        if \"Requested format is not available\" in err:\n",
        "            # Fallback to 'best'\n",
        "            print(f\"⚠️ Format `{fmt}` not found; falling back to `best`.\")\n",
        "            try:\n",
        "                with yt_dlp.YoutubeDL(make_opts(\"best\")) as ydl:\n",
        "                    ydl.download([url])\n",
        "                    info = ydl.extract_info(url, download=False)\n",
        "            except Exception as e2:\n",
        "                return {\n",
        "                    \"success\": False,\n",
        "                    \"error\": f\"Fallback also failed: {type(e2).__name__}: {e2}\",\n",
        "                    \"url\": url,\n",
        "                }\n",
        "        else:\n",
        "            # Other download errors\n",
        "            suggestions = {\n",
        "                \"Video unavailable\": \"May be private, deleted or geo-blocked\",\n",
        "                \"Sign in to confirm your age\": \"Requires age verification\",\n",
        "                \"This live event has ended\": \"Livestream is over\",\n",
        "                \"No video formats found\": \"No downloadable formats\",\n",
        "            }\n",
        "            return {\n",
        "                \"success\": False,\n",
        "                \"error\": err,\n",
        "                \"suggestion\": next(\n",
        "                    (s for k, s in suggestions.items() if k in err),\n",
        "                    \"Check URL and availability\",\n",
        "                ),\n",
        "                \"url\": url,\n",
        "            }\n",
        "\n",
        "    # At this point the file should be downloaded\n",
        "    files = list(Path(output_dir).glob(f\"{info['title']}*.*\"))\n",
        "    if not files:\n",
        "        return {\"success\": False, \"error\": \"File not found after download\", \"url\": url}\n",
        "\n",
        "    path = str(files[0])\n",
        "    size_mb = round(files[0].stat().st_size / (1024**2), 2)\n",
        "\n",
        "    return {\n",
        "        \"success\": True,\n",
        "        \"title\": info.get(\"title\"),\n",
        "        \"file_path\": path,\n",
        "        \"file_size_mb\": size_mb,\n",
        "        \"duration\": info.get(\"duration\"),\n",
        "        \"uploader\": info.get(\"uploader\"),\n",
        "        \"view_count\": info.get(\"view_count\"),\n",
        "        \"url\": url,\n",
        "    }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[youtube] Extracting URL: https://www.youtube.com/watch?v=20DThpeng84\n",
            "[youtube] 20DThpeng84: Downloading webpage\n",
            "[youtube] 20DThpeng84: Downloading tv client config\n",
            "[youtube] 20DThpeng84: Downloading tv player API JSON\n",
            "[youtube] 20DThpeng84: Downloading ios player API JSON\n",
            "[youtube] 20DThpeng84: Downloading m3u8 information\n",
            "⚠️ Video >1h; this may take a while.\n",
            "⬇️ Downloading at format `best[height<=1080]`…\n",
            "[youtube] Extracting URL: https://www.youtube.com/watch?v=20DThpeng84\n",
            "[youtube] 20DThpeng84: Downloading webpage\n",
            "[youtube] 20DThpeng84: Downloading tv client config\n",
            "[youtube] 20DThpeng84: Downloading tv player API JSON\n",
            "[youtube] 20DThpeng84: Downloading ios player API JSON\n",
            "[youtube] 20DThpeng84: Downloading m3u8 information\n",
            "[info] 20DThpeng84: Downloading 1 format(s): 18\n",
            "[download] Destination: videos/FC Barcelona vs Real Betis (5-1) ｜ CDR 2025 Full Match.mp4\n",
            "[download] 100% of  390.84MiB in 00:01:47 at 3.63MiB/s      \n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'success': False,\n",
              " 'error': 'File not found after download',\n",
              " 'url': 'https://www.youtube.com/watch?v=20DThpeng84'}"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Create directory videos if it doesn't exist\n",
        "if not os.path.exists(\"videos\"):\n",
        "    os.makedirs(\"videos\")\n",
        "\n",
        "# Download a video from youtube and save it to the videos directory\n",
        "# Or you can add the video to the videos directory and skip the download step and move to the next step\n",
        "video_url = (\n",
        "    \"https://www.youtube.com/watch?v=20DThpeng84\"  # This is a video of a football match\n",
        ")\n",
        "\n",
        "# Download the video\n",
        "download_youtube_video(video_url)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# You should change the downloaded video name to \"video\" or change the video name below to the title of the stored video\n",
        "video_title = \"video\"\n",
        "\n",
        "# Convert the video to images\n",
        "video_to_images(\n",
        "    video_path=f\"videos/{video_title}.mp4\", output_dir=\"frames\", interval_seconds=2\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 1.2 Setting environment variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "set_env_securely(\"VOYAGE_API_KEY\", \"Enter your Voyage API Key: \")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "set_env_securely(\"OPENAI_API_KEY\", \"Enter your OpenAI API Key: \")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/richmondalake/miniconda3/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "import voyageai\n",
        "from openai import OpenAI\n",
        "\n",
        "# Initialize Voyage AI client\n",
        "# Ensure Voyage API key is set in the environment variables\n",
        "voyageai_client = voyageai.Client()\n",
        "\n",
        "# Initialize OpenAI client\n",
        "# Ensure OpenAI API key is set in the environment variables\n",
        "openai_client = OpenAI()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 1.3 Generating Embeddings with Voyage AI\n",
        "\n",
        "Voyage AI multimodal-3 is a state-of-the-art embedding model that revolutionizes how we process documents containing interleaved text and images by vectorizing both modalities through a unified transformer backbone, eliminating the need for complex document parsing while improving retrieval accuracy by an average of 19.63% over competing models. \n",
        "\n",
        "**Unlike traditional CLIP-based models that process text and images separately, voyage-multimodal-3 captures the contextual relationships between visual and textual elements in screenshots, PDFs, slides, tables, and figures, making it ideal for RAG applications and semantic search across content-rich documents where visual layout and textual content are equally important.**\n",
        "\n",
        "Learn more on Voyage AI Multimodal embeddings here: https://docs.voyageai.com/docs/multimodal-embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import List\n",
        "\n",
        "from PIL import Image\n",
        "\n",
        "\n",
        "def get_voyage_embedding(data: Image.Image | str, input_type: str = \"document\") -> List:\n",
        "    \"\"\"\n",
        "    Get Voyage AI embeddings for images and text.\n",
        "\n",
        "    Args:\n",
        "        data (Image.Image | str): An image or text to embed.\n",
        "        input_type (str): Input type, either \"document\" or \"query\".\n",
        "\n",
        "    Returns:\n",
        "        List: Embeddings as a list.\n",
        "    \"\"\"\n",
        "    embedding = voyageai_client.multimodal_embed(\n",
        "        inputs=[[data]], model=\"voyage-multimodal-3\", input_type=input_type\n",
        "    ).embeddings[0]\n",
        "    return embedding"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [],
      "source": [
        "import base64\n",
        "\n",
        "\n",
        "def encode_image_to_base64(image_path: str) -> str:\n",
        "    \"\"\"\n",
        "    Encode an image file to base64 string for OpenAI.\n",
        "\n",
        "    Args:\n",
        "        image_path (str): Path to the image file\n",
        "\n",
        "    Returns:\n",
        "        str: Base64 encoded image string\n",
        "    \"\"\"\n",
        "    with open(image_path, \"rb\") as image_file:\n",
        "        return base64.b64encode(image_file.read()).decode(\"utf-8\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import List, Optional\n",
        "\n",
        "\n",
        "def get_image_embedding(\n",
        "    image_path: str, input_type: str = \"document\"\n",
        ") -> Optional[List[float]]:\n",
        "    \"\"\"\n",
        "    Get embedding for a single image file using Voyage AI.\n",
        "\n",
        "    Args:\n",
        "        image_path (str): Path to the image file\n",
        "        input_type (str): Input type for embedding (\"document\" or \"query\")\n",
        "\n",
        "    Returns:\n",
        "        Optional[List[float]]: Image embedding vector or None if failed\n",
        "    \"\"\"\n",
        "    try:\n",
        "        # Load image using PIL\n",
        "        image = Image.open(image_path)\n",
        "\n",
        "        # Get embedding using the Voyage AI client\n",
        "        embedding = get_voyage_embedding(image, input_type)\n",
        "\n",
        "        print(\n",
        "            f\"✓ Got embedding for {os.path.basename(image_path)} (dimension: {len(embedding)})\"\n",
        "        )\n",
        "        return embedding\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error getting embedding for {image_path}: {e}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 1.3.1 Generating metadata with each from OpenAI Vision"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [],
      "source": [
        "def generate_frame_description(image_path: str) -> Optional[str]:\n",
        "    \"\"\"\n",
        "    Generate a description of the frame content using OpenAI vision model.\n",
        "\n",
        "    Args:\n",
        "        image_path (str): Path to the image file\n",
        "\n",
        "    Returns:\n",
        "        Optional[str]: Description of the frame content or None if failed\n",
        "    \"\"\"\n",
        "    try:\n",
        "        # Encode image to base64\n",
        "        base64_image = encode_image_to_base64(image_path)\n",
        "\n",
        "        # Create the prompt for frame description\n",
        "        response = openai_client.chat.completions.create(\n",
        "            model=\"gpt-4o\",  # Use GPT-4 vision model\n",
        "            messages=[\n",
        "                {\n",
        "                    \"role\": \"user\",\n",
        "                    \"content\": [\n",
        "                        {\n",
        "                            \"type\": \"text\",\n",
        "                            \"text\": \"Describe what you see in this video frame. Include details about objects, people, actions, setting, and any text visible. Be specific and descriptive but concise.\",\n",
        "                        },\n",
        "                        {\n",
        "                            \"type\": \"image_url\",\n",
        "                            \"image_url\": {\n",
        "                                \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n",
        "                            },\n",
        "                        },\n",
        "                    ],\n",
        "                }\n",
        "            ],\n",
        "            max_tokens=300,\n",
        "        )\n",
        "\n",
        "        description = response.choices[0].message.content\n",
        "        print(f\"✓ Generated description for {os.path.basename(image_path)}\")\n",
        "        return description\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error generating description for {image_path}: {e}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 1.4 Generating frame metadata\n",
        "\n",
        "This code processes video frames to extract both AI embeddings and text descriptions for intelligent search capabilities.\n",
        "\n",
        "**`process_single_frame()`** - Handles individual frame processing:\n",
        "1. Loads an image using PIL\n",
        "2. Generates a vector embedding using Voyage AI (captures visual semantic meaning)\n",
        "3. Creates a text description using OpenAI's vision model\n",
        "4. Returns both as a dictionary or None if processing fails\n",
        "\n",
        "**`process_frames_to_embeddings_with_descriptions()`** - Batch processes all frames:\n",
        "1. **Discovers frames** - Scans the frames directory for image files (.jpg, .png, etc.)\n",
        "2. **Processes sequentially** - Calls `process_single_frame()` for each image\n",
        "3. **Adds metadata** - Extracts frame number and timestamp from filename \n",
        "4. **Rate limiting** - Includes delays between API calls to avoid hitting service limits\n",
        "5. **Progress tracking** - Shows processing status and handles failures gracefully\n",
        "\n",
        "The result is a dictionary mapping each frame filename to its embedding vector, text description, frame number, and timestamp - enabling both visual and semantic search capabilities."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import Dict\n",
        "\n",
        "from PIL import Image\n",
        "\n",
        "\n",
        "def process_single_frame(\n",
        "    image_path: str, input_type: str = \"document\"\n",
        ") -> Optional[Dict]:\n",
        "    \"\"\"\n",
        "    Process a single frame to get both embedding and description.\n",
        "\n",
        "    Args:\n",
        "        image_path (str): Path to the image file\n",
        "        input_type (str): Input type for embedding (\"document\" or \"query\")\n",
        "\n",
        "    Returns:\n",
        "        Optional[Dict]: Dictionary with 'embedding' and 'frame_description' or None if failed\n",
        "    \"\"\"\n",
        "    try:\n",
        "        # Load image using PIL for embedding\n",
        "        image = Image.open(image_path)\n",
        "\n",
        "        # Get embedding using Voyage AI\n",
        "        embedding = get_voyage_embedding(image, input_type)\n",
        "\n",
        "        # Generate description using OpenAI\n",
        "        frame_description = generate_frame_description(image_path)\n",
        "\n",
        "        if embedding is not None and frame_description is not None:\n",
        "            return {\"embedding\": embedding, \"frame_description\": frame_description}\n",
        "        else:\n",
        "            print(f\"Failed to process {image_path} - missing embedding or description\")\n",
        "            return None\n",
        "    except Exception as e:\n",
        "        print(f\"Error processing frame {image_path}: {e}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {},
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "\n",
        "def process_frames_to_embeddings_with_descriptions(\n",
        "    frames_dir: str = \"frames\",\n",
        "    input_type: str = \"document\",\n",
        "    delay_seconds: float = 0.5,\n",
        "    cut_off_frame: float = 300,\n",
        ") -> Dict[str, Dict]:\n",
        "    \"\"\"\n",
        "    Process all images in frames folder and get both Voyage AI embeddings and OpenAI descriptions.\n",
        "\n",
        "    Args:\n",
        "        frames_dir (str): Directory containing frame images (default: \"frames\")\n",
        "        input_type (str): Input type for embeddings (\"document\" or \"query\")\n",
        "        delay_seconds (float): Delay between API calls to avoid rate limits\n",
        "\n",
        "    Returns:\n",
        "        Dict[str, Dict]: Dictionary mapping image filenames to {'embedding': [...], 'frame_description': '...'}\n",
        "    \"\"\"\n",
        "\n",
        "    # Check if frames directory exists\n",
        "    if not os.path.exists(frames_dir):\n",
        "        raise FileNotFoundError(f\"Frames directory '{frames_dir}' not found\")\n",
        "\n",
        "    # Get all image files from the directory\n",
        "    image_extensions = {\".jpg\", \".jpeg\", \".png\", \".bmp\", \".tiff\", \".webp\"}\n",
        "    image_files = []\n",
        "\n",
        "    for file in os.listdir(frames_dir):\n",
        "        if Path(file).suffix.lower() in image_extensions:\n",
        "            image_files.append(os.path.join(frames_dir, file))\n",
        "\n",
        "    if not image_files:\n",
        "        print(f\"No image files found in '{frames_dir}'\")\n",
        "        return {}\n",
        "\n",
        "    print(f\"Found {len(image_files)} images in '{frames_dir}'\")\n",
        "    print(\"Processing frames for embeddings and descriptions...\")\n",
        "\n",
        "    # Process each image\n",
        "    frame_data = {}\n",
        "    failed_count = 0\n",
        "\n",
        "    for i, image_path in enumerate(sorted(image_files), 1):\n",
        "        print(f\"\\nProcessing {i}/{len(image_files)}: {os.path.basename(image_path)}\")\n",
        "\n",
        "        result = process_single_frame(image_path, input_type)\n",
        "\n",
        "        if result is not None:\n",
        "            # Add the frame number to the frame data\n",
        "            result[\"frame_number\"] = i\n",
        "            # Extract the frame timestamp from the image path \"frame_0001_t2.0s.jpg\"\n",
        "            # FIXED: Extract actual timestamp from \"t2.0s\" part, not frame number\n",
        "            filename = os.path.basename(image_path)\n",
        "            try:\n",
        "                # Look for pattern like \"t2.0s\" in filename\n",
        "                # timestamp_part = [part for part in filename.split(\"_\") if part.startswith(\"t\") and part.endswith(\"s.jpg\")][0]\n",
        "                # Extract number from \"t2.0s.jpg\" -> \"2.0\"\n",
        "                timestamp_part = [\n",
        "                    part\n",
        "                    for part in filename.split(\"_\")\n",
        "                    if part.startswith(\"t\") and part.endswith(\"s.jpg\")\n",
        "                ][0]\n",
        "                result[\"frame_timestamp\"] = float(\n",
        "                    timestamp_part[1:-5]\n",
        "                )  # Remove \"t\" and \"s.jpg\"\n",
        "                print(f\"✅ Timestamp parsed successfully: {result['frame_timestamp']}\")\n",
        "            except (IndexError, ValueError) as e:\n",
        "                print(\n",
        "                    f\"⚠️ Could not parse timestamp from {filename}, using frame number: {e}\"\n",
        "                )\n",
        "                # Fallback to frame number if parsing fails\n",
        "                result[\"frame_timestamp\"] = float(i * 2)  # Assume 2-second intervals\n",
        "            frame_data[os.path.basename(image_path)] = result\n",
        "            print(\n",
        "                f\"✓ Complete - Embedding: {len(result['embedding'])}D, Description: {len(result['frame_description'])} chars\"\n",
        "            )\n",
        "        else:\n",
        "            failed_count += 1\n",
        "\n",
        "        # Add delay to respect rate limits (especially for OpenAI)\n",
        "        if i < len(image_files):  # Don't delay after the last image\n",
        "            time.sleep(delay_seconds)\n",
        "\n",
        "        if cut_off_frame is not None and i == cut_off_frame:\n",
        "            break\n",
        "\n",
        "    print(f\"\\n🎉 Completed! Successfully processed {len(frame_data)} frames\")\n",
        "    if failed_count > 0:\n",
        "        print(f\"⚠️ Failed to process {failed_count} frames\")\n",
        "\n",
        "    return frame_data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "frame_data = process_frames_to_embeddings_with_descriptions(\n",
        "    frames_dir=\"frames\", input_type=\"document\", delay_seconds=0.5, cut_off_frame=500\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>embedding</th>\n",
              "      <th>frame_description</th>\n",
              "      <th>frame_number</th>\n",
              "      <th>frame_timestamp</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>frame_0000_t0.0s.jpg</th>\n",
              "      <td>[0.0081787109375, -0.0291748046875, 0.03515625...</td>\n",
              "      <td>The video frame captures an aerial view of a l...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>frame_0001_t2.0s.jpg</th>\n",
              "      <td>[0.003021240234375, -0.048095703125, 0.0500488...</td>\n",
              "      <td>This image shows an aerial view of a large, we...</td>\n",
              "      <td>2</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>frame_0002_t3.9s.jpg</th>\n",
              "      <td>[0.01177978515625, -0.03857421875, 0.044433593...</td>\n",
              "      <td>The image shows a large stadium filled with sp...</td>\n",
              "      <td>3</td>\n",
              "      <td>3.9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>frame_0003_t5.9s.jpg</th>\n",
              "      <td>[0.04541015625, 0.00994873046875, 0.013671875,...</td>\n",
              "      <td>In the video frame, a group of athletes and ch...</td>\n",
              "      <td>4</td>\n",
              "      <td>5.9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>frame_0004_t7.9s.jpg</th>\n",
              "      <td>[0.051025390625, -0.01007080078125, 0.00347900...</td>\n",
              "      <td>The image depicts a group of people, including...</td>\n",
              "      <td>5</td>\n",
              "      <td>7.9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                              embedding  \\\n",
              "frame_0000_t0.0s.jpg  [0.0081787109375, -0.0291748046875, 0.03515625...   \n",
              "frame_0001_t2.0s.jpg  [0.003021240234375, -0.048095703125, 0.0500488...   \n",
              "frame_0002_t3.9s.jpg  [0.01177978515625, -0.03857421875, 0.044433593...   \n",
              "frame_0003_t5.9s.jpg  [0.04541015625, 0.00994873046875, 0.013671875,...   \n",
              "frame_0004_t7.9s.jpg  [0.051025390625, -0.01007080078125, 0.00347900...   \n",
              "\n",
              "                                                      frame_description  \\\n",
              "frame_0000_t0.0s.jpg  The video frame captures an aerial view of a l...   \n",
              "frame_0001_t2.0s.jpg  This image shows an aerial view of a large, we...   \n",
              "frame_0002_t3.9s.jpg  The image shows a large stadium filled with sp...   \n",
              "frame_0003_t5.9s.jpg  In the video frame, a group of athletes and ch...   \n",
              "frame_0004_t7.9s.jpg  The image depicts a group of people, including...   \n",
              "\n",
              "                      frame_number  frame_timestamp  \n",
              "frame_0000_t0.0s.jpg             1              0.0  \n",
              "frame_0001_t2.0s.jpg             2              2.0  \n",
              "frame_0002_t3.9s.jpg             3              3.9  \n",
              "frame_0003_t5.9s.jpg             4              5.9  \n",
              "frame_0004_t7.9s.jpg             5              7.9  "
            ]
          },
          "execution_count": 94,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Convert frame data to a pandas dataframe\n",
        "frame_data_df = pd.DataFrame.from_dict(frame_data, orient=\"index\")\n",
        "\n",
        "frame_data_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 2: Connecting and Saving Data To MongoDB "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {},
      "outputs": [],
      "source": [
        "set_env_securely(\"MONGODB_URI\", \"Enter your MongoDB URI: \")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pymongo\n",
        "\n",
        "\n",
        "def get_mongo_client(mongo_uri):\n",
        "    \"\"\"Establish and validate connection to the MongoDB.\"\"\"\n",
        "\n",
        "    client = pymongo.MongoClient(\n",
        "        mongo_uri, appname=\"devrel.showcase.agents.video_intelligence.python\"\n",
        "    )\n",
        "\n",
        "    # Validate the connection\n",
        "    ping_result = client.admin.command(\"ping\")\n",
        "    if ping_result.get(\"ok\") == 1.0:\n",
        "        # Connection successful\n",
        "        print(\"Connection to MongoDB successful\")\n",
        "        return client\n",
        "    else:\n",
        "        print(\"Connection to MongoDB failed\")\n",
        "    return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Connection to MongoDB successful\n"
          ]
        }
      ],
      "source": [
        "DB_NAME = \"video_intelligence\"\n",
        "db_client = get_mongo_client(os.environ.get(\"MONGODB_URI\"))\n",
        "db = db_client[DB_NAME]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 2.1 Collection creation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Collection names\n",
        "FRAME_INTELLIGENCE_METADATA = \"video_intelligence\"\n",
        "VIDEO_LIBRARY = \"video_library\"\n",
        "PREVIOUS_FRAME_INCIDENTS = \"previous_frame_incidentS\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Embedding dimensions\n",
        "EMBEDDING_DIMENSIONS = 1024"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create collections\n",
        "\n",
        "\n",
        "def create_collections():\n",
        "    existing_collections = db.list_collection_names()\n",
        "    print(f\"Existing collections: {existing_collections}\")\n",
        "\n",
        "    # If the collection does not exist, create it\n",
        "    if FRAME_INTELLIGENCE_METADATA not in existing_collections:\n",
        "        db.create_collection(FRAME_INTELLIGENCE_METADATA)\n",
        "        print(f\"Created collection: {FRAME_INTELLIGENCE_METADATA}\")\n",
        "    else:\n",
        "        print(f\"Collection {FRAME_INTELLIGENCE_METADATA} already exists\")\n",
        "\n",
        "    if VIDEO_LIBRARY not in existing_collections:\n",
        "        db.create_collection(VIDEO_LIBRARY)\n",
        "        print(f\"Created collection: {VIDEO_LIBRARY}\")\n",
        "    else:\n",
        "        print(f\"Collection {VIDEO_LIBRARY} already exists\")\n",
        "\n",
        "    if PREVIOUS_FRAME_INCIDENTS not in existing_collections:\n",
        "        db.create_collection(PREVIOUS_FRAME_INCIDENTS)\n",
        "        print(f\"Created collection: {PREVIOUS_FRAME_INCIDENTS}\")\n",
        "    else:\n",
        "        print(f\"Collection {PREVIOUS_FRAME_INCIDENTS} already exists\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Existing collections: ['video_intelligence', 'video_library', 'previous_frame_incidentS']\n",
            "Collection video_intelligence already exists\n",
            "Collection video_library already exists\n",
            "Collection previous_frame_incidentS already exists\n"
          ]
        }
      ],
      "source": [
        "create_collections()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 2.2 Creating Vector and Search Indexes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create Indexes\n",
        "from pymongo.operations import SearchIndexModel\n",
        "\n",
        "\n",
        "# Create vector search index if it doesn't exist\n",
        "def create_vector_search_index(\n",
        "    collection,\n",
        "    vector_index_name,\n",
        "    dimensions=1024,\n",
        "    quantization=\"scalar\",\n",
        "    embedding_path=\"embedding\",\n",
        "):\n",
        "    # Check if index already exists\n",
        "    try:\n",
        "        existing_indexes = collection.list_search_indexes()\n",
        "        for index in existing_indexes:\n",
        "            if index[\"name\"] == vector_index_name:\n",
        "                print(f\"Vector search index '{vector_index_name}' already exists.\")\n",
        "                return\n",
        "    except Exception as e:\n",
        "        print(f\"Could not list search indexes: {e}\")\n",
        "        return\n",
        "\n",
        "    index_definition = {\n",
        "        \"fields\": [\n",
        "            {\n",
        "                \"type\": \"vector\",\n",
        "                \"path\": embedding_path,\n",
        "                \"numDimensions\": dimensions,\n",
        "                \"similarity\": \"cosine\",\n",
        "            }\n",
        "        ]\n",
        "    }\n",
        "\n",
        "    if quantization == \"scalar\":\n",
        "        index_definition[\"fields\"][0][\"quantization\"] = quantization\n",
        "\n",
        "    if quantization == \"binary\":\n",
        "        index_definition[\"fields\"][0][\"quantization\"] = quantization\n",
        "\n",
        "    # Create vector search index\n",
        "    search_index_model = SearchIndexModel(\n",
        "        definition=index_definition,\n",
        "        name=vector_index_name,\n",
        "        type=\"vectorSearch\",\n",
        "    )\n",
        "\n",
        "    try:\n",
        "        result = collection.create_search_index(model=search_index_model)\n",
        "        print(f\"New search index named '{result}' is building.\")\n",
        "    except Exception as e:\n",
        "        print(f\"Error creating vector search index: {e}\")\n",
        "        return\n",
        "\n",
        "    # Wait for initial sync to complete\n",
        "    print(\n",
        "        f\"Polling to check if the index '{result}' is ready. This may take up to a minute.\"\n",
        "    )\n",
        "    predicate = lambda index: index.get(\"queryable\") is True\n",
        "\n",
        "    while True:\n",
        "        try:\n",
        "            indices = list(collection.list_search_indexes(result))\n",
        "            if indices and predicate(indices[0]):\n",
        "                break\n",
        "            time.sleep(5)\n",
        "        except Exception as e:\n",
        "            print(f\"Error checking index readiness: {e}\")\n",
        "            time.sleep(5)\n",
        "\n",
        "    print(f\"{result} is ready for querying.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This code below defines vector‐search indexes on several collections to support efficient similarity queries over high-dimensional embeddings. \n",
        "\n",
        "For each target collection—such as frame metadata, past incident records, and video libraries—it creates named indexes using varying quantization strategies. \n",
        "\n",
        "Scalar and binary quantization compress embeddings for reduced storage and faster lookups, while full-fidelity indexes preserve maximum precision at the cost of higher resource usage.\n",
        "\n",
        "**By configuring multiple index variants on the same collection, you can benchmark and choose the optimal trade-off between search accuracy, speed, and storage footprint.**\n",
        "**Once built, these indexes enable rapid nearest‐neighbor retrieval of semantically similar items for tasks like incident detection, frame comparison, and content recommendation.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Vector search index 'vector_search_index_scalar' already exists.\n",
            "Vector search index 'vector_search_index_full_fidelity' already exists.\n",
            "Vector search index 'vector_search_index_binary' already exists.\n",
            "Vector search index 'incident_vector_index_scalar' already exists.\n",
            "Vector search index 'video_vector_index' already exists.\n"
          ]
        }
      ],
      "source": [
        "create_vector_search_index(\n",
        "    db[FRAME_INTELLIGENCE_METADATA], \"vector_search_index_scalar\", quantization=\"scalar\"\n",
        ")\n",
        "create_vector_search_index(\n",
        "    db[FRAME_INTELLIGENCE_METADATA],\n",
        "    \"vector_search_index_full_fidelity\",\n",
        "    quantization=\"full_fidelity\",\n",
        ")\n",
        "create_vector_search_index(\n",
        "    db[FRAME_INTELLIGENCE_METADATA],\n",
        "    \"vector_search_index_binary\",\n",
        "    quantization=\"binary\",\n",
        ")\n",
        "create_vector_search_index(\n",
        "    db[PREVIOUS_FRAME_INCIDENTS], \"incident_vector_index_scalar\", quantization=\"scalar\"\n",
        ")\n",
        "create_vector_search_index(\n",
        "    db[VIDEO_LIBRARY], \"video_vector_index\", quantization=\"full_fidelity\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The code below is a helper wraps MongoDB Atlas Search index creation: \n",
        "- given a collection, an index-definition dict, and a name, \n",
        "- it builds a SearchIndexModel, calls create_search_index, \n",
        "- and returns the result—printing success or catching errors and returning None.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {},
      "outputs": [],
      "source": [
        "def create_text_search_index(collection, index_definition, index_name):\n",
        "    \"\"\"\n",
        "    Create a search index for a MongoDB Atlas collection.\n",
        "\n",
        "    Args:\n",
        "    collection: MongoDB collection object\n",
        "    index_definition: Dictionary defining the index mappings\n",
        "    index_name: String name for the index\n",
        "\n",
        "    Returns:\n",
        "    str: Result of the index creation operation\n",
        "    \"\"\"\n",
        "\n",
        "    try:\n",
        "        search_index_model = SearchIndexModel(\n",
        "            definition=index_definition, name=index_name\n",
        "        )\n",
        "\n",
        "        result = collection.create_search_index(model=search_index_model)\n",
        "        print(f\"Search index '{index_name}' created successfully\")\n",
        "        return result\n",
        "    except Exception as e:\n",
        "        print(f\"Error creating search index: {e!s}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {},
      "outputs": [],
      "source": [
        "frame_intelligence_index_definition = {\n",
        "    \"mappings\": {\n",
        "        \"dynamic\": True,\n",
        "        \"fields\": {\n",
        "            # \"video_id\": {\n",
        "            #     \"type\": \"string\",\n",
        "            # },\n",
        "            # \"video_name\": {\n",
        "            #     \"type\": \"string\",\n",
        "            # },\n",
        "            # \"video_url\": {\n",
        "            #     \"type\": \"string\",\n",
        "            # },\n",
        "            \"frame_description\": {\n",
        "                \"type\": \"string\",\n",
        "            },\n",
        "            \"frame_number\": {\n",
        "                \"type\": \"number\",\n",
        "            },\n",
        "            \"frame_timestamp\": {\n",
        "                \"type\": \"date\",\n",
        "            },\n",
        "        },\n",
        "    }\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Search index 'frame_intelligence_index' created successfully\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'frame_intelligence_index'"
            ]
          },
          "execution_count": 106,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "create_text_search_index(\n",
        "    db[FRAME_INTELLIGENCE_METADATA],\n",
        "    frame_intelligence_index_definition,\n",
        "    \"frame_intelligence_index\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 3: Data Ingestion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The step starts by clearing out any existing documents in the three target collections (`FRAME_INTELLIGENCE_METADATA`, `PREVIOUS_FRAME_INCIDENTS`, and `VIDEO_LIBRARY`) via repeated calls to `delete_many({})`, ensuring you’re working with a clean slate before seeding new data.\n",
        "\n",
        "Next, it converts your Pandas DataFrame (`frame_data_df`) into a list of Python dictionaries with `to_dict(orient=\"records\")`, then uses `insert_many` on the `frame_intelligence_collection` (aliased from `db[FRAME_INTELLIGENCE_METADATA]`) to bulk-load those records. \n",
        "\n",
        "This pattern guarantees that your frame intelligence collection is freshly populated and ready for downstream tasks like vector indexing or semantic search.\n",
        "\n",
        "**Because there’s no additional transformation pipeline—no ETL steps, schema migrations, or data-wrangling utilities—loading new data is straightforward.**\n",
        "**You simply clear, convert, and insert, which keeps the setup simple and minimizes the chance of errors or mismatches between your source DataFrame and the MongoDB collection.**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DeleteResult({'n': 0, 'electionId': ObjectId('7fffffff0000000000000002'), 'opTime': {'ts': Timestamp(1751369841, 50), 't': 2}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1751369841, 50), 'signature': {'hash': b'Ln\\xd3a\\x05\\xc94u\\xe5\\x05\\xb3\\x1b5\\x15\\xcd%\\xdb\\x854b', 'keyId': 7520068280199938053}}, 'operationTime': Timestamp(1751369841, 50)}, acknowledged=True)"
            ]
          },
          "execution_count": 107,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Ensure the collections are empty\n",
        "db[FRAME_INTELLIGENCE_METADATA].delete_many({})\n",
        "db[PREVIOUS_FRAME_INCIDENTS].delete_many({})\n",
        "db[VIDEO_LIBRARY].delete_many({})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "InsertManyResult([ObjectId('6863c872a9040ef0b5adcb35'), ObjectId('6863c872a9040ef0b5adcb36'), ObjectId('6863c872a9040ef0b5adcb37'), ObjectId('6863c872a9040ef0b5adcb38'), ObjectId('6863c872a9040ef0b5adcb39'), ObjectId('6863c872a9040ef0b5adcb3a'), ObjectId('6863c872a9040ef0b5adcb3b'), ObjectId('6863c872a9040ef0b5adcb3c'), ObjectId('6863c872a9040ef0b5adcb3d'), ObjectId('6863c872a9040ef0b5adcb3e'), ObjectId('6863c872a9040ef0b5adcb3f'), ObjectId('6863c872a9040ef0b5adcb40'), ObjectId('6863c872a9040ef0b5adcb41'), ObjectId('6863c872a9040ef0b5adcb42'), ObjectId('6863c872a9040ef0b5adcb43'), ObjectId('6863c872a9040ef0b5adcb44'), ObjectId('6863c872a9040ef0b5adcb45'), ObjectId('6863c872a9040ef0b5adcb46'), ObjectId('6863c872a9040ef0b5adcb47'), ObjectId('6863c872a9040ef0b5adcb48'), ObjectId('6863c872a9040ef0b5adcb49'), ObjectId('6863c872a9040ef0b5adcb4a'), ObjectId('6863c872a9040ef0b5adcb4b'), ObjectId('6863c872a9040ef0b5adcb4c'), ObjectId('6863c872a9040ef0b5adcb4d'), ObjectId('6863c872a9040ef0b5adcb4e'), ObjectId('6863c872a9040ef0b5adcb4f'), ObjectId('6863c872a9040ef0b5adcb50'), ObjectId('6863c872a9040ef0b5adcb51'), ObjectId('6863c872a9040ef0b5adcb52'), ObjectId('6863c872a9040ef0b5adcb53'), ObjectId('6863c872a9040ef0b5adcb54'), ObjectId('6863c872a9040ef0b5adcb55'), ObjectId('6863c872a9040ef0b5adcb56'), ObjectId('6863c872a9040ef0b5adcb57'), ObjectId('6863c872a9040ef0b5adcb58'), ObjectId('6863c872a9040ef0b5adcb59'), ObjectId('6863c872a9040ef0b5adcb5a'), ObjectId('6863c872a9040ef0b5adcb5b'), ObjectId('6863c872a9040ef0b5adcb5c'), ObjectId('6863c872a9040ef0b5adcb5d'), ObjectId('6863c872a9040ef0b5adcb5e'), ObjectId('6863c872a9040ef0b5adcb5f'), ObjectId('6863c872a9040ef0b5adcb60'), ObjectId('6863c872a9040ef0b5adcb61'), ObjectId('6863c872a9040ef0b5adcb62'), ObjectId('6863c872a9040ef0b5adcb63'), ObjectId('6863c872a9040ef0b5adcb64'), ObjectId('6863c872a9040ef0b5adcb65'), ObjectId('6863c872a9040ef0b5adcb66'), ObjectId('6863c872a9040ef0b5adcb67'), ObjectId('6863c872a9040ef0b5adcb68'), ObjectId('6863c872a9040ef0b5adcb69'), ObjectId('6863c872a9040ef0b5adcb6a'), ObjectId('6863c872a9040ef0b5adcb6b'), ObjectId('6863c872a9040ef0b5adcb6c'), ObjectId('6863c872a9040ef0b5adcb6d'), ObjectId('6863c872a9040ef0b5adcb6e'), ObjectId('6863c872a9040ef0b5adcb6f'), ObjectId('6863c872a9040ef0b5adcb70'), ObjectId('6863c872a9040ef0b5adcb71'), ObjectId('6863c872a9040ef0b5adcb72'), ObjectId('6863c872a9040ef0b5adcb73'), ObjectId('6863c872a9040ef0b5adcb74'), ObjectId('6863c872a9040ef0b5adcb75'), ObjectId('6863c872a9040ef0b5adcb76'), ObjectId('6863c872a9040ef0b5adcb77'), ObjectId('6863c872a9040ef0b5adcb78'), ObjectId('6863c872a9040ef0b5adcb79'), ObjectId('6863c872a9040ef0b5adcb7a'), ObjectId('6863c872a9040ef0b5adcb7b'), ObjectId('6863c872a9040ef0b5adcb7c'), ObjectId('6863c872a9040ef0b5adcb7d'), ObjectId('6863c872a9040ef0b5adcb7e'), ObjectId('6863c872a9040ef0b5adcb7f'), ObjectId('6863c872a9040ef0b5adcb80'), ObjectId('6863c872a9040ef0b5adcb81'), ObjectId('6863c872a9040ef0b5adcb82'), ObjectId('6863c872a9040ef0b5adcb83'), ObjectId('6863c872a9040ef0b5adcb84'), ObjectId('6863c872a9040ef0b5adcb85'), ObjectId('6863c872a9040ef0b5adcb86'), ObjectId('6863c872a9040ef0b5adcb87'), ObjectId('6863c872a9040ef0b5adcb88'), ObjectId('6863c872a9040ef0b5adcb89'), ObjectId('6863c872a9040ef0b5adcb8a'), ObjectId('6863c872a9040ef0b5adcb8b'), ObjectId('6863c872a9040ef0b5adcb8c'), ObjectId('6863c872a9040ef0b5adcb8d'), ObjectId('6863c872a9040ef0b5adcb8e'), ObjectId('6863c872a9040ef0b5adcb8f'), ObjectId('6863c872a9040ef0b5adcb90'), ObjectId('6863c872a9040ef0b5adcb91'), ObjectId('6863c872a9040ef0b5adcb92'), ObjectId('6863c872a9040ef0b5adcb93'), ObjectId('6863c872a9040ef0b5adcb94'), ObjectId('6863c872a9040ef0b5adcb95'), ObjectId('6863c872a9040ef0b5adcb96'), ObjectId('6863c872a9040ef0b5adcb97'), ObjectId('6863c872a9040ef0b5adcb98'), ObjectId('6863c872a9040ef0b5adcb99'), ObjectId('6863c872a9040ef0b5adcb9a'), ObjectId('6863c872a9040ef0b5adcb9b'), ObjectId('6863c872a9040ef0b5adcb9c'), ObjectId('6863c872a9040ef0b5adcb9d'), ObjectId('6863c872a9040ef0b5adcb9e'), ObjectId('6863c872a9040ef0b5adcb9f'), ObjectId('6863c872a9040ef0b5adcba0'), ObjectId('6863c872a9040ef0b5adcba1'), ObjectId('6863c872a9040ef0b5adcba2'), ObjectId('6863c872a9040ef0b5adcba3'), ObjectId('6863c872a9040ef0b5adcba4'), ObjectId('6863c872a9040ef0b5adcba5'), ObjectId('6863c872a9040ef0b5adcba6'), ObjectId('6863c872a9040ef0b5adcba7'), ObjectId('6863c872a9040ef0b5adcba8'), ObjectId('6863c872a9040ef0b5adcba9'), ObjectId('6863c872a9040ef0b5adcbaa'), ObjectId('6863c872a9040ef0b5adcbab'), ObjectId('6863c872a9040ef0b5adcbac'), ObjectId('6863c872a9040ef0b5adcbad'), ObjectId('6863c872a9040ef0b5adcbae'), ObjectId('6863c872a9040ef0b5adcbaf'), ObjectId('6863c872a9040ef0b5adcbb0'), ObjectId('6863c872a9040ef0b5adcbb1'), ObjectId('6863c872a9040ef0b5adcbb2'), ObjectId('6863c872a9040ef0b5adcbb3'), ObjectId('6863c872a9040ef0b5adcbb4'), ObjectId('6863c872a9040ef0b5adcbb5'), ObjectId('6863c872a9040ef0b5adcbb6'), ObjectId('6863c872a9040ef0b5adcbb7'), ObjectId('6863c872a9040ef0b5adcbb8'), ObjectId('6863c872a9040ef0b5adcbb9'), ObjectId('6863c872a9040ef0b5adcbba'), ObjectId('6863c872a9040ef0b5adcbbb'), ObjectId('6863c872a9040ef0b5adcbbc'), ObjectId('6863c872a9040ef0b5adcbbd'), ObjectId('6863c872a9040ef0b5adcbbe'), ObjectId('6863c872a9040ef0b5adcbbf'), ObjectId('6863c872a9040ef0b5adcbc0'), ObjectId('6863c872a9040ef0b5adcbc1'), ObjectId('6863c872a9040ef0b5adcbc2'), ObjectId('6863c872a9040ef0b5adcbc3'), ObjectId('6863c872a9040ef0b5adcbc4'), ObjectId('6863c872a9040ef0b5adcbc5'), ObjectId('6863c872a9040ef0b5adcbc6'), ObjectId('6863c872a9040ef0b5adcbc7'), ObjectId('6863c872a9040ef0b5adcbc8'), ObjectId('6863c872a9040ef0b5adcbc9'), ObjectId('6863c872a9040ef0b5adcbca'), ObjectId('6863c872a9040ef0b5adcbcb'), ObjectId('6863c872a9040ef0b5adcbcc'), ObjectId('6863c872a9040ef0b5adcbcd'), ObjectId('6863c872a9040ef0b5adcbce'), ObjectId('6863c872a9040ef0b5adcbcf'), ObjectId('6863c872a9040ef0b5adcbd0'), ObjectId('6863c872a9040ef0b5adcbd1'), ObjectId('6863c872a9040ef0b5adcbd2'), ObjectId('6863c872a9040ef0b5adcbd3'), ObjectId('6863c872a9040ef0b5adcbd4'), ObjectId('6863c872a9040ef0b5adcbd5'), ObjectId('6863c872a9040ef0b5adcbd6'), ObjectId('6863c872a9040ef0b5adcbd7'), ObjectId('6863c872a9040ef0b5adcbd8'), ObjectId('6863c872a9040ef0b5adcbd9'), ObjectId('6863c872a9040ef0b5adcbda'), ObjectId('6863c872a9040ef0b5adcbdb'), ObjectId('6863c872a9040ef0b5adcbdc'), ObjectId('6863c872a9040ef0b5adcbdd'), ObjectId('6863c872a9040ef0b5adcbde'), ObjectId('6863c872a9040ef0b5adcbdf'), ObjectId('6863c872a9040ef0b5adcbe0'), ObjectId('6863c872a9040ef0b5adcbe1'), ObjectId('6863c872a9040ef0b5adcbe2'), ObjectId('6863c872a9040ef0b5adcbe3'), ObjectId('6863c872a9040ef0b5adcbe4'), ObjectId('6863c872a9040ef0b5adcbe5'), ObjectId('6863c872a9040ef0b5adcbe6'), ObjectId('6863c872a9040ef0b5adcbe7'), ObjectId('6863c872a9040ef0b5adcbe8'), ObjectId('6863c872a9040ef0b5adcbe9'), ObjectId('6863c872a9040ef0b5adcbea'), ObjectId('6863c872a9040ef0b5adcbeb'), ObjectId('6863c872a9040ef0b5adcbec'), ObjectId('6863c872a9040ef0b5adcbed'), ObjectId('6863c872a9040ef0b5adcbee'), ObjectId('6863c872a9040ef0b5adcbef'), ObjectId('6863c872a9040ef0b5adcbf0'), ObjectId('6863c872a9040ef0b5adcbf1'), ObjectId('6863c872a9040ef0b5adcbf2'), ObjectId('6863c872a9040ef0b5adcbf3'), ObjectId('6863c872a9040ef0b5adcbf4'), ObjectId('6863c872a9040ef0b5adcbf5'), ObjectId('6863c872a9040ef0b5adcbf6'), ObjectId('6863c872a9040ef0b5adcbf7'), ObjectId('6863c872a9040ef0b5adcbf8'), ObjectId('6863c872a9040ef0b5adcbf9'), ObjectId('6863c872a9040ef0b5adcbfa'), ObjectId('6863c872a9040ef0b5adcbfb'), ObjectId('6863c872a9040ef0b5adcbfc'), ObjectId('6863c872a9040ef0b5adcbfd'), ObjectId('6863c872a9040ef0b5adcbfe'), ObjectId('6863c872a9040ef0b5adcbff'), ObjectId('6863c872a9040ef0b5adcc00'), ObjectId('6863c872a9040ef0b5adcc01'), ObjectId('6863c872a9040ef0b5adcc02'), ObjectId('6863c872a9040ef0b5adcc03'), ObjectId('6863c872a9040ef0b5adcc04'), ObjectId('6863c872a9040ef0b5adcc05'), ObjectId('6863c872a9040ef0b5adcc06'), ObjectId('6863c872a9040ef0b5adcc07'), ObjectId('6863c872a9040ef0b5adcc08'), ObjectId('6863c872a9040ef0b5adcc09'), ObjectId('6863c872a9040ef0b5adcc0a'), ObjectId('6863c872a9040ef0b5adcc0b'), ObjectId('6863c872a9040ef0b5adcc0c'), ObjectId('6863c872a9040ef0b5adcc0d'), ObjectId('6863c872a9040ef0b5adcc0e'), ObjectId('6863c872a9040ef0b5adcc0f'), ObjectId('6863c872a9040ef0b5adcc10'), ObjectId('6863c872a9040ef0b5adcc11'), ObjectId('6863c872a9040ef0b5adcc12'), ObjectId('6863c872a9040ef0b5adcc13'), ObjectId('6863c872a9040ef0b5adcc14'), ObjectId('6863c872a9040ef0b5adcc15'), ObjectId('6863c872a9040ef0b5adcc16'), ObjectId('6863c872a9040ef0b5adcc17'), ObjectId('6863c872a9040ef0b5adcc18'), ObjectId('6863c872a9040ef0b5adcc19'), ObjectId('6863c872a9040ef0b5adcc1a'), ObjectId('6863c872a9040ef0b5adcc1b'), ObjectId('6863c872a9040ef0b5adcc1c'), ObjectId('6863c872a9040ef0b5adcc1d'), ObjectId('6863c872a9040ef0b5adcc1e'), ObjectId('6863c872a9040ef0b5adcc1f'), ObjectId('6863c872a9040ef0b5adcc20'), ObjectId('6863c872a9040ef0b5adcc21'), ObjectId('6863c872a9040ef0b5adcc22'), ObjectId('6863c872a9040ef0b5adcc23'), ObjectId('6863c872a9040ef0b5adcc24'), ObjectId('6863c872a9040ef0b5adcc25'), ObjectId('6863c872a9040ef0b5adcc26'), ObjectId('6863c872a9040ef0b5adcc27'), ObjectId('6863c872a9040ef0b5adcc28'), ObjectId('6863c872a9040ef0b5adcc29'), ObjectId('6863c872a9040ef0b5adcc2a'), ObjectId('6863c872a9040ef0b5adcc2b'), ObjectId('6863c872a9040ef0b5adcc2c'), ObjectId('6863c872a9040ef0b5adcc2d'), ObjectId('6863c872a9040ef0b5adcc2e'), ObjectId('6863c872a9040ef0b5adcc2f'), ObjectId('6863c872a9040ef0b5adcc30'), ObjectId('6863c872a9040ef0b5adcc31'), ObjectId('6863c872a9040ef0b5adcc32'), ObjectId('6863c872a9040ef0b5adcc33'), ObjectId('6863c872a9040ef0b5adcc34'), ObjectId('6863c872a9040ef0b5adcc35'), ObjectId('6863c872a9040ef0b5adcc36'), ObjectId('6863c872a9040ef0b5adcc37'), ObjectId('6863c872a9040ef0b5adcc38'), ObjectId('6863c872a9040ef0b5adcc39'), ObjectId('6863c872a9040ef0b5adcc3a'), ObjectId('6863c872a9040ef0b5adcc3b'), ObjectId('6863c872a9040ef0b5adcc3c'), ObjectId('6863c872a9040ef0b5adcc3d'), ObjectId('6863c872a9040ef0b5adcc3e'), ObjectId('6863c872a9040ef0b5adcc3f'), ObjectId('6863c872a9040ef0b5adcc40'), ObjectId('6863c872a9040ef0b5adcc41'), ObjectId('6863c872a9040ef0b5adcc42'), ObjectId('6863c872a9040ef0b5adcc43'), ObjectId('6863c872a9040ef0b5adcc44'), ObjectId('6863c872a9040ef0b5adcc45'), ObjectId('6863c872a9040ef0b5adcc46'), ObjectId('6863c872a9040ef0b5adcc47'), ObjectId('6863c872a9040ef0b5adcc48'), ObjectId('6863c872a9040ef0b5adcc49'), ObjectId('6863c872a9040ef0b5adcc4a'), ObjectId('6863c872a9040ef0b5adcc4b'), ObjectId('6863c872a9040ef0b5adcc4c'), ObjectId('6863c872a9040ef0b5adcc4d'), ObjectId('6863c872a9040ef0b5adcc4e'), ObjectId('6863c872a9040ef0b5adcc4f'), ObjectId('6863c872a9040ef0b5adcc50'), ObjectId('6863c872a9040ef0b5adcc51'), ObjectId('6863c872a9040ef0b5adcc52'), ObjectId('6863c872a9040ef0b5adcc53'), ObjectId('6863c872a9040ef0b5adcc54'), ObjectId('6863c872a9040ef0b5adcc55'), ObjectId('6863c872a9040ef0b5adcc56'), ObjectId('6863c872a9040ef0b5adcc57'), ObjectId('6863c872a9040ef0b5adcc58'), ObjectId('6863c872a9040ef0b5adcc59'), ObjectId('6863c872a9040ef0b5adcc5a'), ObjectId('6863c872a9040ef0b5adcc5b'), ObjectId('6863c872a9040ef0b5adcc5c'), ObjectId('6863c872a9040ef0b5adcc5d'), ObjectId('6863c872a9040ef0b5adcc5e'), ObjectId('6863c872a9040ef0b5adcc5f'), ObjectId('6863c872a9040ef0b5adcc60'), ObjectId('6863c872a9040ef0b5adcc61'), ObjectId('6863c872a9040ef0b5adcc62'), ObjectId('6863c872a9040ef0b5adcc63'), ObjectId('6863c872a9040ef0b5adcc64'), ObjectId('6863c872a9040ef0b5adcc65'), ObjectId('6863c872a9040ef0b5adcc66'), ObjectId('6863c872a9040ef0b5adcc67'), ObjectId('6863c872a9040ef0b5adcc68'), ObjectId('6863c872a9040ef0b5adcc69'), ObjectId('6863c872a9040ef0b5adcc6a'), ObjectId('6863c872a9040ef0b5adcc6b'), ObjectId('6863c872a9040ef0b5adcc6c'), ObjectId('6863c872a9040ef0b5adcc6d'), ObjectId('6863c872a9040ef0b5adcc6e'), ObjectId('6863c872a9040ef0b5adcc6f'), ObjectId('6863c872a9040ef0b5adcc70'), ObjectId('6863c872a9040ef0b5adcc71'), ObjectId('6863c872a9040ef0b5adcc72'), ObjectId('6863c872a9040ef0b5adcc73'), ObjectId('6863c872a9040ef0b5adcc74'), ObjectId('6863c872a9040ef0b5adcc75'), ObjectId('6863c872a9040ef0b5adcc76'), ObjectId('6863c872a9040ef0b5adcc77'), ObjectId('6863c872a9040ef0b5adcc78'), ObjectId('6863c872a9040ef0b5adcc79'), ObjectId('6863c872a9040ef0b5adcc7a'), ObjectId('6863c872a9040ef0b5adcc7b'), ObjectId('6863c872a9040ef0b5adcc7c'), ObjectId('6863c872a9040ef0b5adcc7d'), ObjectId('6863c872a9040ef0b5adcc7e'), ObjectId('6863c872a9040ef0b5adcc7f'), ObjectId('6863c872a9040ef0b5adcc80'), ObjectId('6863c872a9040ef0b5adcc81'), ObjectId('6863c872a9040ef0b5adcc82'), ObjectId('6863c872a9040ef0b5adcc83'), ObjectId('6863c872a9040ef0b5adcc84'), ObjectId('6863c872a9040ef0b5adcc85'), ObjectId('6863c872a9040ef0b5adcc86'), ObjectId('6863c872a9040ef0b5adcc87'), ObjectId('6863c872a9040ef0b5adcc88'), ObjectId('6863c872a9040ef0b5adcc89'), ObjectId('6863c872a9040ef0b5adcc8a'), ObjectId('6863c872a9040ef0b5adcc8b'), ObjectId('6863c872a9040ef0b5adcc8c'), ObjectId('6863c872a9040ef0b5adcc8d'), ObjectId('6863c872a9040ef0b5adcc8e'), ObjectId('6863c872a9040ef0b5adcc8f'), ObjectId('6863c872a9040ef0b5adcc90'), ObjectId('6863c872a9040ef0b5adcc91'), ObjectId('6863c872a9040ef0b5adcc92'), ObjectId('6863c872a9040ef0b5adcc93'), ObjectId('6863c872a9040ef0b5adcc94'), ObjectId('6863c872a9040ef0b5adcc95'), ObjectId('6863c872a9040ef0b5adcc96'), ObjectId('6863c872a9040ef0b5adcc97'), ObjectId('6863c872a9040ef0b5adcc98'), ObjectId('6863c872a9040ef0b5adcc99'), ObjectId('6863c872a9040ef0b5adcc9a'), ObjectId('6863c872a9040ef0b5adcc9b'), ObjectId('6863c872a9040ef0b5adcc9c'), ObjectId('6863c872a9040ef0b5adcc9d'), ObjectId('6863c872a9040ef0b5adcc9e'), ObjectId('6863c872a9040ef0b5adcc9f'), ObjectId('6863c872a9040ef0b5adcca0'), ObjectId('6863c872a9040ef0b5adcca1'), ObjectId('6863c872a9040ef0b5adcca2'), ObjectId('6863c872a9040ef0b5adcca3'), ObjectId('6863c872a9040ef0b5adcca4'), ObjectId('6863c872a9040ef0b5adcca5'), ObjectId('6863c872a9040ef0b5adcca6'), ObjectId('6863c872a9040ef0b5adcca7'), ObjectId('6863c872a9040ef0b5adcca8'), ObjectId('6863c872a9040ef0b5adcca9'), ObjectId('6863c872a9040ef0b5adccaa'), ObjectId('6863c872a9040ef0b5adccab'), ObjectId('6863c872a9040ef0b5adccac'), ObjectId('6863c872a9040ef0b5adccad'), ObjectId('6863c872a9040ef0b5adccae'), ObjectId('6863c872a9040ef0b5adccaf'), ObjectId('6863c872a9040ef0b5adccb0'), ObjectId('6863c872a9040ef0b5adccb1'), ObjectId('6863c872a9040ef0b5adccb2'), ObjectId('6863c872a9040ef0b5adccb3'), ObjectId('6863c872a9040ef0b5adccb4'), ObjectId('6863c872a9040ef0b5adccb5'), ObjectId('6863c872a9040ef0b5adccb6'), ObjectId('6863c872a9040ef0b5adccb7'), ObjectId('6863c872a9040ef0b5adccb8'), ObjectId('6863c872a9040ef0b5adccb9'), ObjectId('6863c872a9040ef0b5adccba'), ObjectId('6863c872a9040ef0b5adccbb'), ObjectId('6863c872a9040ef0b5adccbc'), ObjectId('6863c872a9040ef0b5adccbd'), ObjectId('6863c872a9040ef0b5adccbe'), ObjectId('6863c872a9040ef0b5adccbf'), ObjectId('6863c872a9040ef0b5adccc0'), ObjectId('6863c872a9040ef0b5adccc1'), ObjectId('6863c872a9040ef0b5adccc2'), ObjectId('6863c872a9040ef0b5adccc3'), ObjectId('6863c872a9040ef0b5adccc4'), ObjectId('6863c872a9040ef0b5adccc5'), ObjectId('6863c872a9040ef0b5adccc6'), ObjectId('6863c872a9040ef0b5adccc7'), ObjectId('6863c872a9040ef0b5adccc8'), ObjectId('6863c872a9040ef0b5adccc9'), ObjectId('6863c872a9040ef0b5adccca'), ObjectId('6863c872a9040ef0b5adcccb'), ObjectId('6863c872a9040ef0b5adcccc'), ObjectId('6863c872a9040ef0b5adcccd'), ObjectId('6863c872a9040ef0b5adccce'), ObjectId('6863c872a9040ef0b5adcccf'), ObjectId('6863c872a9040ef0b5adccd0'), ObjectId('6863c872a9040ef0b5adccd1'), ObjectId('6863c872a9040ef0b5adccd2'), ObjectId('6863c872a9040ef0b5adccd3'), ObjectId('6863c872a9040ef0b5adccd4'), ObjectId('6863c872a9040ef0b5adccd5'), ObjectId('6863c872a9040ef0b5adccd6'), ObjectId('6863c872a9040ef0b5adccd7'), ObjectId('6863c872a9040ef0b5adccd8'), ObjectId('6863c872a9040ef0b5adccd9'), ObjectId('6863c872a9040ef0b5adccda'), ObjectId('6863c872a9040ef0b5adccdb'), ObjectId('6863c872a9040ef0b5adccdc'), ObjectId('6863c872a9040ef0b5adccdd'), ObjectId('6863c872a9040ef0b5adccde'), ObjectId('6863c872a9040ef0b5adccdf'), ObjectId('6863c872a9040ef0b5adcce0'), ObjectId('6863c872a9040ef0b5adcce1'), ObjectId('6863c872a9040ef0b5adcce2'), ObjectId('6863c872a9040ef0b5adcce3'), ObjectId('6863c872a9040ef0b5adcce4'), ObjectId('6863c872a9040ef0b5adcce5'), ObjectId('6863c872a9040ef0b5adcce6'), ObjectId('6863c872a9040ef0b5adcce7'), ObjectId('6863c872a9040ef0b5adcce8'), ObjectId('6863c872a9040ef0b5adcce9'), ObjectId('6863c872a9040ef0b5adccea'), ObjectId('6863c872a9040ef0b5adcceb'), ObjectId('6863c872a9040ef0b5adccec'), ObjectId('6863c872a9040ef0b5adcced'), ObjectId('6863c872a9040ef0b5adccee'), ObjectId('6863c872a9040ef0b5adccef'), ObjectId('6863c872a9040ef0b5adccf0'), ObjectId('6863c872a9040ef0b5adccf1'), ObjectId('6863c872a9040ef0b5adccf2'), ObjectId('6863c872a9040ef0b5adccf3'), ObjectId('6863c872a9040ef0b5adccf4'), ObjectId('6863c872a9040ef0b5adccf5'), ObjectId('6863c872a9040ef0b5adccf6'), ObjectId('6863c872a9040ef0b5adccf7'), ObjectId('6863c872a9040ef0b5adccf8'), ObjectId('6863c872a9040ef0b5adccf9'), ObjectId('6863c872a9040ef0b5adccfa'), ObjectId('6863c872a9040ef0b5adccfb'), ObjectId('6863c872a9040ef0b5adccfc'), ObjectId('6863c872a9040ef0b5adccfd'), ObjectId('6863c872a9040ef0b5adccfe'), ObjectId('6863c872a9040ef0b5adccff'), ObjectId('6863c872a9040ef0b5adcd00'), ObjectId('6863c872a9040ef0b5adcd01'), ObjectId('6863c872a9040ef0b5adcd02'), ObjectId('6863c872a9040ef0b5adcd03'), ObjectId('6863c872a9040ef0b5adcd04'), ObjectId('6863c872a9040ef0b5adcd05'), ObjectId('6863c872a9040ef0b5adcd06'), ObjectId('6863c872a9040ef0b5adcd07'), ObjectId('6863c872a9040ef0b5adcd08'), ObjectId('6863c872a9040ef0b5adcd09'), ObjectId('6863c872a9040ef0b5adcd0a'), ObjectId('6863c872a9040ef0b5adcd0b'), ObjectId('6863c872a9040ef0b5adcd0c'), ObjectId('6863c872a9040ef0b5adcd0d'), ObjectId('6863c872a9040ef0b5adcd0e'), ObjectId('6863c872a9040ef0b5adcd0f'), ObjectId('6863c872a9040ef0b5adcd10'), ObjectId('6863c872a9040ef0b5adcd11'), ObjectId('6863c872a9040ef0b5adcd12'), ObjectId('6863c872a9040ef0b5adcd13'), ObjectId('6863c872a9040ef0b5adcd14'), ObjectId('6863c872a9040ef0b5adcd15'), ObjectId('6863c872a9040ef0b5adcd16'), ObjectId('6863c872a9040ef0b5adcd17'), ObjectId('6863c872a9040ef0b5adcd18'), ObjectId('6863c872a9040ef0b5adcd19'), ObjectId('6863c872a9040ef0b5adcd1a'), ObjectId('6863c872a9040ef0b5adcd1b'), ObjectId('6863c872a9040ef0b5adcd1c'), ObjectId('6863c872a9040ef0b5adcd1d'), ObjectId('6863c872a9040ef0b5adcd1e'), ObjectId('6863c872a9040ef0b5adcd1f'), ObjectId('6863c872a9040ef0b5adcd20'), ObjectId('6863c872a9040ef0b5adcd21'), ObjectId('6863c872a9040ef0b5adcd22'), ObjectId('6863c872a9040ef0b5adcd23'), ObjectId('6863c872a9040ef0b5adcd24'), ObjectId('6863c872a9040ef0b5adcd25'), ObjectId('6863c872a9040ef0b5adcd26'), ObjectId('6863c872a9040ef0b5adcd27'), ObjectId('6863c872a9040ef0b5adcd28')], acknowledged=True)"
            ]
          },
          "execution_count": 108,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "frame_intelligence_documents = frame_data_df.to_dict(orient=\"records\")\n",
        "\n",
        "# Create a new collection for frame intelligence\n",
        "frame_intelligence_collection = db[FRAME_INTELLIGENCE_METADATA]\n",
        "\n",
        "# Insert the frame intelligence documents into the collection\n",
        "frame_intelligence_collection.insert_many(frame_intelligence_documents)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 4: Retrieval Methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 4.1 Semantic Search powered by Vector Search"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the code below **`semantic_search_with_mongodb`** wraps the end-to-end process of running a semantic vector search in MongoDB Atlas. It first obtains a numeric embedding for the user’s query via `get_voyage_embedding`, then constructs a two-stage aggregation pipeline:\n",
        "\n",
        "1. A **`$vectorSearch`** stage that leverages your precreated vector index to find semantically similar documents.\n",
        "2. A **`$project`** stage that strips out the raw embedding and internal `_id`, and injects the similarity score (`vectorSearchScore`) into each result.\n",
        "   Finally, it executes the pipeline and returns the top-N results as a Python list, abstracting away all of the boilerplate needed to perform high-precision, retrieval-grounded queries.\n",
        "\n",
        "Under the hood, the MongoDB **`$vectorSearch`** operator supports several key parameters for tuning accuracy and performance:\n",
        "\n",
        "* **`index`** (string): the name of the vector index to use. ([mongodb.com][1])\n",
        "* **`queryVector`** (array): the embedding representing the query text. ([mongodb.com][1])\n",
        "* **`path`** (string): the document field that stores precomputed embeddings. ([mongodb.com][1])\n",
        "* **`numCandidates`** (int): how many nearest-neighbor candidates to retrieve before final scoring—higher values improve recall at the cost of latency. ([mongodb.com][2])\n",
        "* **`limit`** (int): the maximum number of top-scoring documents to return. ([mongodb.com][1])\n",
        "\n",
        "By tuning `numCandidates` and `limit`, you can balance throughput, resource usage, and retrieval fidelity for your specific dataset.\n",
        "\n",
        "[1]: https://www.mongodb.com/docs/drivers/rust/v3.1/fundamentals/aggregation/vector-search/ \"Atlas Vector Search - Rust Driver v3.1 - MongoDB Docs\"\n",
        "[2]: https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/ \"Run Vector Search Queries - Atlas - MongoDB Docs\"\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Implementing vector search\n",
        "def semantic_search_with_mongodb(\n",
        "    user_query, collection, top_n=5, vector_search_index_name=\"vector_search_index\"\n",
        "):\n",
        "    \"\"\"\n",
        "    Perform a vector search in the MongoDB collection based on the user query.\n",
        "\n",
        "    Args:\n",
        "    user_query (str): The user's query string.\n",
        "    collection (MongoCollection): The MongoDB collection to search.\n",
        "    top_n (int): The number of top results to return.\n",
        "    vector_search_index_name (str): The name of the vector search index.\n",
        "\n",
        "    Returns:\n",
        "    list: A list of matching documents.\n",
        "    \"\"\"\n",
        "\n",
        "    # Retrieve the pre-generated embedding for the query from our dictionary\n",
        "    # This embedding represents the semantic meaning of the query as a vector\n",
        "    query_embedding = get_voyage_embedding(user_query, input_type=\"query\")\n",
        "\n",
        "    # Check if we have a valid embedding for the query\n",
        "    if query_embedding is None:\n",
        "        return \"Invalid query or embedding generation failed.\"\n",
        "\n",
        "    # Define the vector search stage using MongoDB's $vectorSearch operator\n",
        "    # This stage performs the semantic similarity search\n",
        "    vector_search_stage = {\n",
        "        \"$vectorSearch\": {\n",
        "            \"index\": vector_search_index_name,  # The vector index we created earlier\n",
        "            \"queryVector\": query_embedding,  # The numerical vector representing our query\n",
        "            \"path\": \"embedding\",  # The field containing document embeddings\n",
        "            \"numCandidates\": 100,  # Explore this many vectors for potential matches\n",
        "            \"limit\": top_n,  # Return only the top N most similar results\n",
        "        }\n",
        "    }\n",
        "\n",
        "    # Define which fields to include in the results and their format\n",
        "    project_stage = {\n",
        "        \"$project\": {\n",
        "            \"_id\": 0,  # Exclude MongoDB's internal ID\n",
        "            \"embedding\": 0,\n",
        "            \"score\": {\n",
        "                \"$meta\": \"vectorSearchScore\"  # Include similarity score from vector search\n",
        "            },\n",
        "        }\n",
        "    }\n",
        "\n",
        "    # Combine the search and projection stages into a complete pipeline\n",
        "    pipeline = [vector_search_stage, project_stage]\n",
        "\n",
        "    # Execute the pipeline against our collection and get results\n",
        "    results = collection.aggregate(pipeline)\n",
        "\n",
        "    # Convert cursor to a Python list for easier handling\n",
        "    return list(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The cells below runs the same semantic search query:\n",
        "“Can you get me the frame with the referee on the screen”\n",
        "against three different vector‐search index configurations (`scalar`, `full_fidelity`, and `binary`) on the `FRAME_INTELLIGENCE_METADATA` collection. \n",
        "\n",
        "Each call to `semantic_search_with_mongodb` embeds the user query, invokes the specified index via MongoDB’s `$vectorSearch`, and returns the top 5 most similar frame documents for that quantization strategy.\n",
        "\n",
        "**By assigning the results to `scalar_results`, `full_fidelity_results`, and `binary_results`, you can directly compare how each index type affects retrieval quality and performance. This makes it easy to benchmark and choose the optimal trade-off between precision, speed, and storage footprint for your frame‐matching application.**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {},
      "outputs": [],
      "source": [
        "user_query = \"Can you get me the frame with the refree on the screen\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "metadata": {},
      "outputs": [],
      "source": [
        "scalar_results = semantic_search_with_mongodb(\n",
        "    user_query=user_query,\n",
        "    collection=db[FRAME_INTELLIGENCE_METADATA],\n",
        "    top_n=5,\n",
        "    vector_search_index_name=\"vector_search_index_scalar\",\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'frame_description': 'The frame shows a soccer match in progress. A player wearing a pink and navy blue jersey with a crest and a sponsor logo is sitting on the ground. Another player, wearing a white jersey, is partially in the foreground, obscuring some of the sitting player. The scoreboard at the top left corner displays \"RMA 1 - 0 FCB\" with 11:27 as the time. The ESPN and ESPN+ LIVE logos are visible in the top right corner. The background is red, suggesting an advertisement board.',\n",
              "  'frame_number': 487,\n",
              "  'frame_timestamp': 956.8,\n",
              "  'score': 0.6657900810241699},\n",
              " {'frame_description': 'The video frame shows a soccer match in progress. A referee wearing a bright yellow shirt and black shorts is walking on the field. The referee\\'s shirt has several sponsor logos, including \"Würth\" and \"Hankook.\" The playing field is green grass. \\n\\nOn the top left of the frame, the scoreboard indicates \"RMA 1-0 FCB,\" showing that RMA is leading FCB. Next to the scoreboard, the time displayed is \"07:07.\" In the top right corner, the broadcast logos \"ESPN\" and \"ABC Live\" are visible. \\n\\nAnother person, likely a player, is partially visible in the lower part of the frame.',\n",
              "  'frame_number': 355,\n",
              "  'frame_timestamp': 696.9,\n",
              "  'score': 0.6579937934875488},\n",
              " {'frame_description': 'In the video frame, two soccer players are on a field. \\n\\n- The player on the left is wearing a white jersey with black stripes and the number 22, named \"Rüdiger.\"\\n- The player on the right is wearing a blue and red striped jersey.\\n- Both players appear to be in motion, likely during a match.\\n  \\nThe setting appears to be a professional soccer match broadcast on ESPN and ABC, as indicated by the logos in the top-right corner. \\n\\nThe scoreboard at the top-left corner shows \"RMA 0 - 0 FCB\" with a time of \"02:25,\" suggesting the match is in the early stages. The background shows a blurred advertisement board.',\n",
              "  'frame_number': 212,\n",
              "  'frame_timestamp': 415.4,\n",
              "  'score': 0.6577053070068359},\n",
              " {'frame_description': 'In the video frame, there is a crowded stadium with spectators filling the stands, indicating a live sports event. In the foreground is a large digital overlay. On the left side of the overlay, there is a vertical bar with text reading \"THIBAUT COURTOIS\" and the number \"1,\" suggesting it\\'s displaying player information. On the right side, there\\'s a faint image of a person, and above it is the word \"GOAL.\" The top right corner displays the ESPN logo alongside \"LIVE.\" The setting is vibrant with stadium lights illuminating the scene.',\n",
              "  'frame_number': 70,\n",
              "  'frame_timestamp': 135.8,\n",
              "  'score': 0.6571376323699951},\n",
              " {'frame_description': 'The image shows a soccer match scene. In the foreground, there are three players wearing jerseys. The player on the left, with \"VALVERDE\" on the back of the jersey, has the number 9. He is facing away from the camera. The player in the middle is wearing a blue goalkeeper kit and appears to be looking towards the play. The player on the right is wearing a white jersey with the logo of \"Emirates Fly Better\" visible. Behind them is a soccer net, and the background is filled with spectators.\\n\\nIn the top left corner, there is a scoreboard displaying \"RMA 0 - 0 FCB\" and a time of \"03:56\". The channel logo \"ESPN\" is visible in the top right corner.',\n",
              "  'frame_number': 258,\n",
              "  'frame_timestamp': 505.9,\n",
              "  'score': 0.6536464691162109}]"
            ]
          },
          "execution_count": 112,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scalar_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'frame_description': 'The frame shows a soccer match in progress. A player wearing a pink and navy blue jersey with a crest and a sponsor logo is sitting on the ground. Another player, wearing a white jersey, is partially in the foreground, obscuring some of the sitting player. The scoreboard at the top left corner displays \"RMA 1 - 0 FCB\" with 11:27 as the time. The ESPN and ESPN+ LIVE logos are visible in the top right corner. The background is red, suggesting an advertisement board.',\n",
              "  'frame_number': 487,\n",
              "  'frame_timestamp': 956.8,\n",
              "  'score': 0.6655800938606262},\n",
              " {'frame_description': 'The video frame shows a soccer match in progress. A referee wearing a bright yellow shirt and black shorts is walking on the field. The referee\\'s shirt has several sponsor logos, including \"Würth\" and \"Hankook.\" The playing field is green grass. \\n\\nOn the top left of the frame, the scoreboard indicates \"RMA 1-0 FCB,\" showing that RMA is leading FCB. Next to the scoreboard, the time displayed is \"07:07.\" In the top right corner, the broadcast logos \"ESPN\" and \"ABC Live\" are visible. \\n\\nAnother person, likely a player, is partially visible in the lower part of the frame.',\n",
              "  'frame_number': 355,\n",
              "  'frame_timestamp': 696.9,\n",
              "  'score': 0.6579287052154541},\n",
              " {'frame_description': 'In the video frame, two soccer players are on a field. \\n\\n- The player on the left is wearing a white jersey with black stripes and the number 22, named \"Rüdiger.\"\\n- The player on the right is wearing a blue and red striped jersey.\\n- Both players appear to be in motion, likely during a match.\\n  \\nThe setting appears to be a professional soccer match broadcast on ESPN and ABC, as indicated by the logos in the top-right corner. \\n\\nThe scoreboard at the top-left corner shows \"RMA 0 - 0 FCB\" with a time of \"02:25,\" suggesting the match is in the early stages. The background shows a blurred advertisement board.',\n",
              "  'frame_number': 212,\n",
              "  'frame_timestamp': 415.4,\n",
              "  'score': 0.6575652360916138},\n",
              " {'frame_description': 'In the video frame, there is a crowded stadium with spectators filling the stands, indicating a live sports event. In the foreground is a large digital overlay. On the left side of the overlay, there is a vertical bar with text reading \"THIBAUT COURTOIS\" and the number \"1,\" suggesting it\\'s displaying player information. On the right side, there\\'s a faint image of a person, and above it is the word \"GOAL.\" The top right corner displays the ESPN logo alongside \"LIVE.\" The setting is vibrant with stadium lights illuminating the scene.',\n",
              "  'frame_number': 70,\n",
              "  'frame_timestamp': 135.8,\n",
              "  'score': 0.6569483280181885},\n",
              " {'frame_description': 'The image shows a soccer match scene. In the foreground, there are three players wearing jerseys. The player on the left, with \"VALVERDE\" on the back of the jersey, has the number 9. He is facing away from the camera. The player in the middle is wearing a blue goalkeeper kit and appears to be looking towards the play. The player on the right is wearing a white jersey with the logo of \"Emirates Fly Better\" visible. Behind them is a soccer net, and the background is filled with spectators.\\n\\nIn the top left corner, there is a scoreboard displaying \"RMA 0 - 0 FCB\" and a time of \"03:56\". The channel logo \"ESPN\" is visible in the top right corner.',\n",
              "  'frame_number': 258,\n",
              "  'frame_timestamp': 505.9,\n",
              "  'score': 0.6542713642120361}]"
            ]
          },
          "execution_count": 113,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "full_fidelity_results = semantic_search_with_mongodb(\n",
        "    user_query=user_query,\n",
        "    collection=db[FRAME_INTELLIGENCE_METADATA],\n",
        "    top_n=5,\n",
        "    vector_search_index_name=\"vector_search_index_full_fidelity\",\n",
        ")\n",
        "\n",
        "full_fidelity_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'frame_description': 'The frame shows a soccer match in progress. A player wearing a pink and navy blue jersey with a crest and a sponsor logo is sitting on the ground. Another player, wearing a white jersey, is partially in the foreground, obscuring some of the sitting player. The scoreboard at the top left corner displays \"RMA 1 - 0 FCB\" with 11:27 as the time. The ESPN and ESPN+ LIVE logos are visible in the top right corner. The background is red, suggesting an advertisement board.',\n",
              "  'frame_number': 487,\n",
              "  'frame_timestamp': 956.8,\n",
              "  'score': 0.6655800938606262},\n",
              " {'frame_description': 'The video frame shows a soccer match in progress. A referee wearing a bright yellow shirt and black shorts is walking on the field. The referee\\'s shirt has several sponsor logos, including \"Würth\" and \"Hankook.\" The playing field is green grass. \\n\\nOn the top left of the frame, the scoreboard indicates \"RMA 1-0 FCB,\" showing that RMA is leading FCB. Next to the scoreboard, the time displayed is \"07:07.\" In the top right corner, the broadcast logos \"ESPN\" and \"ABC Live\" are visible. \\n\\nAnother person, likely a player, is partially visible in the lower part of the frame.',\n",
              "  'frame_number': 355,\n",
              "  'frame_timestamp': 696.9,\n",
              "  'score': 0.6579287052154541},\n",
              " {'frame_description': 'The image shows a soccer match in progress. Two players are in the frame, both wearing a dark blue and red striped kit. One player, with the number 6 on his back, is partially seen. The other player is facing away, with visible bandaging on his wrist.\\n\\nThe scoreboard graphic at the top shows \"RMA 1 - 0 FCB\" with a timer reading \"09:23.\" The broadcast logos for \"ESPN\" and \"ABC Live\" are also visible at the top right corner. The playing field has green grass, and a white line is visible in the background indicating part of the pitch markings.',\n",
              "  'frame_number': 424,\n",
              "  'frame_timestamp': 832.7,\n",
              "  'score': 0.6527595520019531},\n",
              " {'frame_description': 'The image captures a moment from a soccer match shown on ESPN. The top left displays a score box indicating \"RMA 1 - 0 FCB\" at 08:51. A player wearing a jersey with pink and navy colors is partially visible. There\\'s a white circular patch on the jersey\\'s sleeve. A graphic partially obscures the player. The setting appears to be a football field with a white boundary line and green grass in the background. The ESPN and ABC Live logos are shown in the top right corner.',\n",
              "  'frame_number': 408,\n",
              "  'frame_timestamp': 801.2,\n",
              "  'score': 0.6490817070007324},\n",
              " {'frame_description': 'The image shows a soccer player in motion on a field, wearing a blue and red jersey with a club crest on the chest. He has black hair and is looking to the side. He is wearing a white wristband on his left hand. In the background, there is a blurred advertisement board. \\n\\nAt the top of the frame, a scoreboard displays \"RMA 1 - 0 FCB\" and a time of \"09:21\". There is also a logo that says \"ESPN\" with the word \"LIVE\" next to it in the top right corner.',\n",
              "  'frame_number': 423,\n",
              "  'frame_timestamp': 830.8,\n",
              "  'score': 0.6490310430526733}]"
            ]
          },
          "execution_count": 114,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "binary_results = semantic_search_with_mongodb(\n",
        "    user_query=user_query,\n",
        "    collection=db[FRAME_INTELLIGENCE_METADATA],\n",
        "    top_n=5,\n",
        "    vector_search_index_name=\"vector_search_index_binary\",\n",
        ")\n",
        "\n",
        "binary_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 4.2 Hybrid Search (Text + Vector Search)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**`hybrid_search`** combines semantic vector search and traditional text search in MongoDB using the `$rankFusion` operator. It first converts the `user_query` into an embedding via `get_voyage_embedding`, then defines two sub-pipelines—one using `$vectorSearch` on the specified `vector_search_index_name`, the other using Atlas Search’s `$search` on `text_search_index_name`. These pipelines each retrieve up to 20 candidates, which are then merged and re-ranked according to specified weights, producing a unified list of the top-`top_n` results enriched with detailed scoring information.\n",
        "\n",
        "The `$rankFusion` stage supports key parameters for fine-tuning relevance blending:\n",
        "\n",
        "* **`pipelines`**: maps names (“vectorPipeline”, “textPipeline”) to aggregation pipelines that source vector and text matches.\n",
        "* **`combination.weights`**: assigns relative importance to each pipeline (e.g. `vector_weight=0.7`, `text_weight=0.3`).\n",
        "* **`scoreDetails`**: when set to `true`, includes per-pipeline scores in each document’s `scoreDetails` field.\n",
        "  After fusion, a `$project` stage hides raw embeddings and internal IDs while surfacing score breakdowns, and a final `$limit` ensures only the top-scoring documents are returned. This abstraction lets you call `hybrid_search(query, collection)` to effortlessly leverage both semantic and lexical matching in one go.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 115,
      "metadata": {},
      "outputs": [],
      "source": [
        "def hybrid_search(\n",
        "    user_query,\n",
        "    collection,\n",
        "    top_n=5,\n",
        "    vector_search_index_name=\"vector_search_index_scalar\",\n",
        "    text_search_index_name=\"text_search_index\",\n",
        "    vector_weight=0.7,\n",
        "    text_weight=0.3,\n",
        "):\n",
        "    \"\"\"\n",
        "    Perform hybrid search using both vector and text search with MongoDB RankFusion.\n",
        "\n",
        "    Args:\n",
        "        user_query (str): The user's query or search term.\n",
        "        collection (Collection): MongoDB collection object.\n",
        "        top_n (int): Number of results to return.\n",
        "        vector_search_index_name (str): Name of the vector search index.\n",
        "        text_search_index_name (str): Name of the text search index.\n",
        "        vector_weight (float): Weight for vector search results (0.0-1.0).\n",
        "        text_weight (float): Weight for text search results (0.0-1.0).\n",
        "\n",
        "    Returns:\n",
        "        List[Dict]: List of search results with scores and details.\n",
        "    \"\"\"\n",
        "\n",
        "    # Convert user query to embedding for vector search\n",
        "    query_embedding = get_voyage_embedding(user_query, input_type=\"query\")\n",
        "\n",
        "    # Build the RankFusion aggregation pipeline\n",
        "    rank_fusion_stage = {\n",
        "        \"$rankFusion\": {\n",
        "            \"input\": {\n",
        "                \"pipelines\": {\n",
        "                    \"vectorPipeline\": [\n",
        "                        {\n",
        "                            \"$vectorSearch\": {\n",
        "                                \"index\": vector_search_index_name,\n",
        "                                \"path\": \"embedding\",\n",
        "                                \"queryVector\": query_embedding,\n",
        "                                \"numCandidates\": 100,\n",
        "                                \"limit\": 20,\n",
        "                            }\n",
        "                        }\n",
        "                    ],\n",
        "                    \"textPipeline\": [\n",
        "                        {\n",
        "                            \"$search\": {\n",
        "                                \"index\": text_search_index_name,\n",
        "                                \"phrase\": {\n",
        "                                    \"query\": user_query,\n",
        "                                    \"path\": \"frame_description\",\n",
        "                                },\n",
        "                            }\n",
        "                        },\n",
        "                        {\"$limit\": 20},\n",
        "                    ],\n",
        "                }\n",
        "            },\n",
        "            \"combination\": {\n",
        "                \"weights\": {\n",
        "                    \"vectorPipeline\": vector_weight,\n",
        "                    \"textPipeline\": text_weight,\n",
        "                }\n",
        "            },\n",
        "            \"scoreDetails\": True,\n",
        "        }\n",
        "    }\n",
        "\n",
        "    # Project stage to select desired fields and include score details\n",
        "    project_stage = {\n",
        "        \"$project\": {\n",
        "            \"_id\": 0,\n",
        "            \"embedding\": 0,\n",
        "            \"scoreDetails\": {\"$meta\": \"scoreDetails\"},\n",
        "        }\n",
        "    }\n",
        "\n",
        "    # Final limit stage\n",
        "    limit_stage = {\"$limit\": top_n}\n",
        "\n",
        "    # Combine all stages into the complete aggregation pipeline\n",
        "    pipeline = [rank_fusion_stage, project_stage, limit_stage]\n",
        "\n",
        "    try:\n",
        "        # Execute the pipeline against the collection\n",
        "        results = list(collection.aggregate(pipeline))\n",
        "\n",
        "        print(f\"Found {len(results)} results for query: '{user_query}'\")\n",
        "\n",
        "        return results\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error executing hybrid search: {e}\")\n",
        "        return []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 116,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Found 5 results for query: 'Can you get me the frame with the refree on the screen'\n"
          ]
        }
      ],
      "source": [
        "scalar_hybrid_search_results = hybrid_search(\n",
        "    user_query=user_query,\n",
        "    collection=db[FRAME_INTELLIGENCE_METADATA],\n",
        "    top_n=5,\n",
        "    vector_search_index_name=\"vector_search_index_scalar\",\n",
        "    text_search_index_name=\"text_search_index\",\n",
        "    vector_weight=0.5,\n",
        "    text_weight=0.5,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'frame_description': 'The frame shows a soccer match in progress. A player wearing a pink and navy blue jersey with a crest and a sponsor logo is sitting on the ground. Another player, wearing a white jersey, is partially in the foreground, obscuring some of the sitting player. The scoreboard at the top left corner displays \"RMA 1 - 0 FCB\" with 11:27 as the time. The ESPN and ESPN+ LIVE logos are visible in the top right corner. The background is red, suggesting an advertisement board.',\n",
              "  'frame_number': 487,\n",
              "  'frame_timestamp': 956.8,\n",
              "  'scoreDetails': {'value': 0.00819672131147541,\n",
              "   'description': 'value output by reciprocal rank fusion algorithm, computed as sum of (weight * (1 / (60 + rank))) across input pipelines from which this document is output, from:',\n",
              "   'details': [{'inputPipelineName': 'textPipeline', 'rank': 0, 'weight': 0.5},\n",
              "    {'inputPipelineName': 'vectorPipeline',\n",
              "     'rank': 1,\n",
              "     'weight': 0.5,\n",
              "     'value': 0.6657900810241699,\n",
              "     'details': []}]}},\n",
              " {'frame_description': 'The video frame shows a soccer match in progress. A referee wearing a bright yellow shirt and black shorts is walking on the field. The referee\\'s shirt has several sponsor logos, including \"Würth\" and \"Hankook.\" The playing field is green grass. \\n\\nOn the top left of the frame, the scoreboard indicates \"RMA 1-0 FCB,\" showing that RMA is leading FCB. Next to the scoreboard, the time displayed is \"07:07.\" In the top right corner, the broadcast logos \"ESPN\" and \"ABC Live\" are visible. \\n\\nAnother person, likely a player, is partially visible in the lower part of the frame.',\n",
              "  'frame_number': 355,\n",
              "  'frame_timestamp': 696.9,\n",
              "  'scoreDetails': {'value': 0.008064516129032258,\n",
              "   'description': 'value output by reciprocal rank fusion algorithm, computed as sum of (weight * (1 / (60 + rank))) across input pipelines from which this document is output, from:',\n",
              "   'details': [{'inputPipelineName': 'textPipeline', 'rank': 0, 'weight': 0.5},\n",
              "    {'inputPipelineName': 'vectorPipeline',\n",
              "     'rank': 2,\n",
              "     'weight': 0.5,\n",
              "     'value': 0.6579937934875488,\n",
              "     'details': []}]}},\n",
              " {'frame_description': 'In the video frame, two soccer players are on a field. \\n\\n- The player on the left is wearing a white jersey with black stripes and the number 22, named \"Rüdiger.\"\\n- The player on the right is wearing a blue and red striped jersey.\\n- Both players appear to be in motion, likely during a match.\\n  \\nThe setting appears to be a professional soccer match broadcast on ESPN and ABC, as indicated by the logos in the top-right corner. \\n\\nThe scoreboard at the top-left corner shows \"RMA 0 - 0 FCB\" with a time of \"02:25,\" suggesting the match is in the early stages. The background shows a blurred advertisement board.',\n",
              "  'frame_number': 212,\n",
              "  'frame_timestamp': 415.4,\n",
              "  'scoreDetails': {'value': 0.007936507936507936,\n",
              "   'description': 'value output by reciprocal rank fusion algorithm, computed as sum of (weight * (1 / (60 + rank))) across input pipelines from which this document is output, from:',\n",
              "   'details': [{'inputPipelineName': 'textPipeline', 'rank': 0, 'weight': 0.5},\n",
              "    {'inputPipelineName': 'vectorPipeline',\n",
              "     'rank': 3,\n",
              "     'weight': 0.5,\n",
              "     'value': 0.6577053070068359,\n",
              "     'details': []}]}},\n",
              " {'frame_description': 'In the video frame, there is a crowded stadium with spectators filling the stands, indicating a live sports event. In the foreground is a large digital overlay. On the left side of the overlay, there is a vertical bar with text reading \"THIBAUT COURTOIS\" and the number \"1,\" suggesting it\\'s displaying player information. On the right side, there\\'s a faint image of a person, and above it is the word \"GOAL.\" The top right corner displays the ESPN logo alongside \"LIVE.\" The setting is vibrant with stadium lights illuminating the scene.',\n",
              "  'frame_number': 70,\n",
              "  'frame_timestamp': 135.8,\n",
              "  'scoreDetails': {'value': 0.0078125,\n",
              "   'description': 'value output by reciprocal rank fusion algorithm, computed as sum of (weight * (1 / (60 + rank))) across input pipelines from which this document is output, from:',\n",
              "   'details': [{'inputPipelineName': 'textPipeline', 'rank': 0, 'weight': 0.5},\n",
              "    {'inputPipelineName': 'vectorPipeline',\n",
              "     'rank': 4,\n",
              "     'weight': 0.5,\n",
              "     'value': 0.6571376323699951,\n",
              "     'details': []}]}},\n",
              " {'frame_description': 'The image shows a soccer match scene. In the foreground, there are three players wearing jerseys. The player on the left, with \"VALVERDE\" on the back of the jersey, has the number 9. He is facing away from the camera. The player in the middle is wearing a blue goalkeeper kit and appears to be looking towards the play. The player on the right is wearing a white jersey with the logo of \"Emirates Fly Better\" visible. Behind them is a soccer net, and the background is filled with spectators.\\n\\nIn the top left corner, there is a scoreboard displaying \"RMA 0 - 0 FCB\" and a time of \"03:56\". The channel logo \"ESPN\" is visible in the top right corner.',\n",
              "  'frame_number': 258,\n",
              "  'frame_timestamp': 505.9,\n",
              "  'scoreDetails': {'value': 0.007692307692307693,\n",
              "   'description': 'value output by reciprocal rank fusion algorithm, computed as sum of (weight * (1 / (60 + rank))) across input pipelines from which this document is output, from:',\n",
              "   'details': [{'inputPipelineName': 'textPipeline', 'rank': 0, 'weight': 0.5},\n",
              "    {'inputPipelineName': 'vectorPipeline',\n",
              "     'rank': 5,\n",
              "     'weight': 0.5,\n",
              "     'value': 0.6536464691162109,\n",
              "     'details': []}]}}]"
            ]
          },
          "execution_count": 117,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scalar_hybrid_search_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 4.3 Viewing the video player and returned time stamp\n",
        "\n",
        "This code creates an interactive video player for Jupyter notebooks that enables intelligent scene navigation based on AI search results. The `create_video_player_with_scenes()` function takes a video file and search results (containing timestamps, descriptions, and similarity scores), then generates an HTML interface with an embedded video player. It automatically handles video encoding by converting smaller files (under 50MB) to base64 for direct embedding, while serving larger videos from their local path with appropriate MIME type detection.\n",
        "\n",
        "The interface features a standard HTML5 video player with custom scene navigation controls below it. Users can click timestamp buttons to instantly jump to specific scenes, with each button showing the frame number, timestamp, similarity score, and description preview. When selected, the full scene description appears above the player, the video jumps to that timestamp, and playback begins automatically. The system includes keyboard shortcuts (spacebar for play/pause, arrow keys for 10-second navigation) and visual feedback effects, creating a seamless experience for exploring video content based on AI-generated embeddings and making it ideal for video analysis and content search."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 118,
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import Dict, List\n",
        "\n",
        "from IPython.display import HTML, display\n",
        "\n",
        "\n",
        "def create_video_player_with_scenes(\n",
        "    video_path: str,\n",
        "    search_results: List[Dict],\n",
        "    user_query: str = \"\",\n",
        "    width: int = 800,\n",
        "    height: int = 450,\n",
        ") -> None:\n",
        "    \"\"\"\n",
        "    Create an interactive video player with scene navigation for Jupyter notebooks.\n",
        "\n",
        "    Args:\n",
        "        video_path (str): Path to the video file\n",
        "        search_results (List[Dict]): Search results with timestamps and descriptions\n",
        "        user_query (str): The original search query that generated these results\n",
        "        width (int): Video player width in pixels\n",
        "        height (int): Video player height in pixels\n",
        "    \"\"\"\n",
        "\n",
        "    if not os.path.exists(video_path):\n",
        "        print(f\"❌ Video file not found: {video_path}\")\n",
        "        return\n",
        "\n",
        "    if not search_results:\n",
        "        print(\"❌ No search results provided\")\n",
        "        return\n",
        "\n",
        "    # Convert video to base64 for embedding (for small videos)\n",
        "    # For large videos, you might want to serve via a local server\n",
        "    video_base64 = None\n",
        "    file_size_mb = os.path.getsize(video_path) / (1024 * 1024)\n",
        "\n",
        "    if file_size_mb < 50:  # Only embed videos smaller than 50MB\n",
        "        with open(video_path, \"rb\") as video_file:\n",
        "            video_data = video_file.read()\n",
        "            video_base64 = base64.b64encode(video_data).decode()\n",
        "\n",
        "    # Get video file extension for MIME type\n",
        "    file_ext = os.path.splitext(video_path)[1].lower()\n",
        "    mime_types = {\n",
        "        \".mp4\": \"video/mp4\",\n",
        "        \".webm\": \"video/webm\",\n",
        "        \".ogg\": \"video/ogg\",\n",
        "        \".avi\": \"video/mp4\",  # Fallback\n",
        "        \".mov\": \"video/mp4\",  # Fallback\n",
        "    }\n",
        "    video_mime = mime_types.get(file_ext, \"video/mp4\")\n",
        "\n",
        "    # Sort search results by timestamp\n",
        "    sorted_results = sorted(search_results, key=lambda x: x.get(\"frame_timestamp\", 0))\n",
        "\n",
        "    # Create HTML with embedded video player and controls\n",
        "    html_content = f\"\"\"\n",
        "    <div style=\"font-family: Arial, sans-serif; max-width: {width + 50}px;\">\n",
        "        <h3>🎬 Video Scene Navigator</h3>\n",
        "        \n",
        "        <!-- Search Query Display -->\n",
        "        {f'''<div style=\"background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 15px; border-radius: 8px; margin-bottom: 20px; box-shadow: 0 2px 10px rgba(0,0,0,0.1);\">\n",
        "            <div style=\"display: flex; align-items: center; gap: 10px;\">\n",
        "                <span style=\"font-size: 18px;\">🔍</span>\n",
        "                <div>\n",
        "                    <div style=\"font-size: 14px; opacity: 0.9; margin-bottom: 5px;\">Search Query:</div>\n",
        "                    <div style=\"font-size: 16px; font-weight: bold;\">\"{user_query}\"</div>\n",
        "                    <div style=\"font-size: 12px; opacity: 0.8; margin-top: 5px;\">Found {len(sorted_results)} matching scenes</div>\n",
        "                </div>\n",
        "            </div>\n",
        "        </div>''' if user_query else ''}\n",
        "        \n",
        "        <!-- Video Player -->\n",
        "        <div style=\"margin-bottom: 20px; border: 2px solid #ddd; border-radius: 8px; overflow: hidden;\">\n",
        "            <video id=\"videoPlayer\" width=\"{width}\" height=\"{height}\" controls style=\"display: block;\">\n",
        "                {\"<source src='data:\" + video_mime + \";base64,\" + video_base64 + \"' type='\" + video_mime + \"'>\" if video_base64 else \"<source src='\" + video_path + \"' type='\" + video_mime + \"'>\" }\n",
        "                Your browser does not support the video tag.\n",
        "            </video>\n",
        "        </div>\n",
        "        \n",
        "        <!-- Current Scene Info -->\n",
        "        <div id=\"currentScene\" style=\"background: #f0f8ff; padding: 15px; border-radius: 8px; margin-bottom: 20px; border-left: 4px solid #4CAF50;\">\n",
        "            <h4 style=\"margin: 0 0 10px 0; color: #333;\">📍 Current Scene</h4>\n",
        "            <p id=\"sceneDescription\" style=\"margin: 0; color: #666; font-style: italic;\">Click a timestamp below to view scene details</p>\n",
        "        </div>\n",
        "        \n",
        "        <!-- Scene Navigation Buttons -->\n",
        "        <div style=\"background: #f9f9f9; padding: 20px; border-radius: 8px;\">\n",
        "            <h4 style=\"margin: 0 0 15px 0; color: #333;\">🎯 Jump to Scenes</h4>\n",
        "            <div style=\"display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); gap: 10px;\">\n",
        "\"\"\"\n",
        "\n",
        "    # Add scene buttons\n",
        "    for i, result in enumerate(sorted_results):\n",
        "        timestamp = result.get(\"frame_timestamp\", 0)\n",
        "        description = result.get(\"frame_description\", \"No description\")\n",
        "        score = result.get(\"score\", 0)\n",
        "        frame_number = result.get(\"frame_number\", 0)\n",
        "\n",
        "        # Truncate description for button\n",
        "        short_desc = description[:60] + \"...\" if len(description) > 60 else description\n",
        "\n",
        "        html_content += f\"\"\"\n",
        "                <button onclick=\"jumpToScene({timestamp}, `{description.replace('`', \"'\").replace('\"', \"'\")}`, {score}, {frame_number})\" \n",
        "                        style=\"padding: 10px; border: 1px solid #ddd; border-radius: 5px; background: white; cursor: pointer; text-align: left; transition: all 0.3s;\"\n",
        "                        onmouseover=\"this.style.background='#e3f2fd'; this.style.borderColor='#2196F3';\"\n",
        "                        onmouseout=\"this.style.background='white'; this.style.borderColor='#ddd';\">\n",
        "                    <div style=\"font-weight: bold; color: #1976D2; margin-bottom: 5px;\">\n",
        "                        ⏱️ {timestamp}s (Frame {frame_number}) | Score: {score:.3f}\n",
        "                    </div>\n",
        "                    <div style=\"font-size: 12px; color: #666;\">\n",
        "                        {short_desc}\n",
        "                    </div>\n",
        "                </button>\n",
        "\"\"\"\n",
        "\n",
        "    # Add JavaScript functionality\n",
        "    html_content += \"\"\"\n",
        "            </div>\n",
        "        </div>\n",
        "        \n",
        "        <!-- Video Controls Info -->\n",
        "        <div style=\"margin-top: 20px; padding: 15px; background: #fff3cd; border-radius: 8px; border-left: 4px solid #ffc107;\">\n",
        "            <h4 style=\"margin: 0 0 10px 0; color: #856404;\">💡 How to Use</h4>\n",
        "            <ul style=\"margin: 0; color: #856404; font-size: 14px;\">\n",
        "                <li>Click any timestamp button to jump to that scene</li>\n",
        "                <li>Use video controls to play, pause, and adjust volume</li>\n",
        "                <li>Scene descriptions appear above when you select a timestamp</li>\n",
        "            </ul>\n",
        "        </div>\n",
        "    </div>\n",
        "\n",
        "    <script>\n",
        "        function jumpToScene(timestamp, description, score, frameNumber) {\n",
        "            const video = document.getElementById('videoPlayer');\n",
        "            const sceneDesc = document.getElementById('sceneDescription');\n",
        "            \n",
        "            // Jump to timestamp\n",
        "            video.currentTime = timestamp;\n",
        "            \n",
        "            // Update scene description\n",
        "            sceneDesc.innerHTML = `\n",
        "                <div style=\"margin-bottom: 10px;\">\n",
        "                    <strong>🎬 Frame ${frameNumber} at ${timestamp}s (Score: ${score.toFixed(3)})</strong>\n",
        "                </div>\n",
        "                <div style=\"line-height: 1.5;\">\n",
        "                    ${description}\n",
        "                </div>\n",
        "            `;\n",
        "            \n",
        "            // Auto-play if paused\n",
        "            if (video.paused) {\n",
        "                video.play().catch(e => console.log('Auto-play prevented by browser'));\n",
        "            }\n",
        "            \n",
        "            // Scroll video into view\n",
        "            video.scrollIntoView({ behavior: 'smooth', block: 'nearest' });\n",
        "        }\n",
        "        \n",
        "        // Add time update listener to show current time\n",
        "        document.getElementById('videoPlayer').addEventListener('timeupdate', function() {\n",
        "            const currentTime = this.currentTime;\n",
        "            // You could add real-time scene detection here\n",
        "        });\n",
        "        \n",
        "        // Add keyboard shortcuts\n",
        "        document.addEventListener('keydown', function(e) {\n",
        "            const video = document.getElementById('videoPlayer');\n",
        "            \n",
        "            switch(e.key) {\n",
        "                case ' ':  // Spacebar to play/pause\n",
        "                    e.preventDefault();\n",
        "                    if (video.paused) {\n",
        "                        video.play();\n",
        "                    } else {\n",
        "                        video.pause();\n",
        "                    }\n",
        "                    break;\n",
        "                case 'ArrowLeft':  // Left arrow to go back 10s\n",
        "                    e.preventDefault();\n",
        "                    video.currentTime = Math.max(0, video.currentTime - 10);\n",
        "                    break;\n",
        "                case 'ArrowRight':  // Right arrow to go forward 10s\n",
        "                    e.preventDefault();\n",
        "                    video.currentTime = Math.min(video.duration, video.currentTime + 10);\n",
        "                    break;\n",
        "            }\n",
        "        });\n",
        "    </script>\n",
        "    \"\"\"\n",
        "\n",
        "    # Display the HTML\n",
        "    display(HTML(html_content))\n",
        "\n",
        "    print(f\"🎬 Video player created with {len(sorted_results)} scenes\")\n",
        "    print(f\"📁 Video: {os.path.basename(video_path)}\")\n",
        "    if file_size_mb >= 50:\n",
        "        print(\"⚠️  Large video file - serving from local path\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 120,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🎬 Creating interactive video player...\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"font-family: Arial, sans-serif; max-width: 850px;\">\n",
              "        <h3>🎬 Video Scene Navigator</h3>\n",
              "        \n",
              "        <!-- Search Query Display -->\n",
              "        <div style=\"background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 15px; border-radius: 8px; margin-bottom: 20px; box-shadow: 0 2px 10px rgba(0,0,0,0.1);\">\n",
              "            <div style=\"display: flex; align-items: center; gap: 10px;\">\n",
              "                <span style=\"font-size: 18px;\">🔍</span>\n",
              "                <div>\n",
              "                    <div style=\"font-size: 14px; opacity: 0.9; margin-bottom: 5px;\">Search Query:</div>\n",
              "                    <div style=\"font-size: 16px; font-weight: bold;\">\"Get me a scene where a player is injured or on the ground\"</div>\n",
              "                    <div style=\"font-size: 12px; opacity: 0.8; margin-top: 5px;\">Found 5 matching scenes</div>\n",
              "                </div>\n",
              "            </div>\n",
              "        </div>\n",
              "        \n",
              "        <!-- Video Player -->\n",
              "        <div style=\"margin-bottom: 20px; border: 2px solid #ddd; border-radius: 8px; overflow: hidden;\">\n",
              "            <video id=\"videoPlayer\" width=\"800\" height=\"450\" controls style=\"display: block;\">\n",
              "                <source src='videos/video.mp4' type='video/mp4'>\n",
              "                Your browser does not support the video tag.\n",
              "            </video>\n",
              "        </div>\n",
              "        \n",
              "        <!-- Current Scene Info -->\n",
              "        <div id=\"currentScene\" style=\"background: #f0f8ff; padding: 15px; border-radius: 8px; margin-bottom: 20px; border-left: 4px solid #4CAF50;\">\n",
              "            <h4 style=\"margin: 0 0 10px 0; color: #333;\">📍 Current Scene</h4>\n",
              "            <p id=\"sceneDescription\" style=\"margin: 0; color: #666; font-style: italic;\">Click a timestamp below to view scene details</p>\n",
              "        </div>\n",
              "        \n",
              "        <!-- Scene Navigation Buttons -->\n",
              "        <div style=\"background: #f9f9f9; padding: 20px; border-radius: 8px;\">\n",
              "            <h4 style=\"margin: 0 0 15px 0; color: #333;\">🎯 Jump to Scenes</h4>\n",
              "            <div style=\"display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); gap: 10px;\">\n",
              "\n",
              "                <button onclick=\"jumpToScene(135.8, `In the video frame, there is a crowded stadium with spectators filling the stands, indicating a live sports event. In the foreground is a large digital overlay. On the left side of the overlay, there is a vertical bar with text reading 'THIBAUT COURTOIS' and the number '1,' suggesting it's displaying player information. On the right side, there's a faint image of a person, and above it is the word 'GOAL.' The top right corner displays the ESPN logo alongside 'LIVE.' The setting is vibrant with stadium lights illuminating the scene.`, 0.6569483280181885, 70)\" \n",
              "                        style=\"padding: 10px; border: 1px solid #ddd; border-radius: 5px; background: white; cursor: pointer; text-align: left; transition: all 0.3s;\"\n",
              "                        onmouseover=\"this.style.background='#e3f2fd'; this.style.borderColor='#2196F3';\"\n",
              "                        onmouseout=\"this.style.background='white'; this.style.borderColor='#ddd';\">\n",
              "                    <div style=\"font-weight: bold; color: #1976D2; margin-bottom: 5px;\">\n",
              "                        ⏱️ 135.8s (Frame 70) | Score: 0.657\n",
              "                    </div>\n",
              "                    <div style=\"font-size: 12px; color: #666;\">\n",
              "                        In the video frame, there is a crowded stadium with spectato...\n",
              "                    </div>\n",
              "                </button>\n",
              "\n",
              "                <button onclick=\"jumpToScene(415.4, `In the video frame, two soccer players are on a field. \n",
              "\n",
              "- The player on the left is wearing a white jersey with black stripes and the number 22, named 'Rüdiger.'\n",
              "- The player on the right is wearing a blue and red striped jersey.\n",
              "- Both players appear to be in motion, likely during a match.\n",
              "  \n",
              "The setting appears to be a professional soccer match broadcast on ESPN and ABC, as indicated by the logos in the top-right corner. \n",
              "\n",
              "The scoreboard at the top-left corner shows 'RMA 0 - 0 FCB' with a time of '02:25,' suggesting the match is in the early stages. The background shows a blurred advertisement board.`, 0.6575652360916138, 212)\" \n",
              "                        style=\"padding: 10px; border: 1px solid #ddd; border-radius: 5px; background: white; cursor: pointer; text-align: left; transition: all 0.3s;\"\n",
              "                        onmouseover=\"this.style.background='#e3f2fd'; this.style.borderColor='#2196F3';\"\n",
              "                        onmouseout=\"this.style.background='white'; this.style.borderColor='#ddd';\">\n",
              "                    <div style=\"font-weight: bold; color: #1976D2; margin-bottom: 5px;\">\n",
              "                        ⏱️ 415.4s (Frame 212) | Score: 0.658\n",
              "                    </div>\n",
              "                    <div style=\"font-size: 12px; color: #666;\">\n",
              "                        In the video frame, two soccer players are on a field. \n",
              "\n",
              "- T...\n",
              "                    </div>\n",
              "                </button>\n",
              "\n",
              "                <button onclick=\"jumpToScene(505.9, `The image shows a soccer match scene. In the foreground, there are three players wearing jerseys. The player on the left, with 'VALVERDE' on the back of the jersey, has the number 9. He is facing away from the camera. The player in the middle is wearing a blue goalkeeper kit and appears to be looking towards the play. The player on the right is wearing a white jersey with the logo of 'Emirates Fly Better' visible. Behind them is a soccer net, and the background is filled with spectators.\n",
              "\n",
              "In the top left corner, there is a scoreboard displaying 'RMA 0 - 0 FCB' and a time of '03:56'. The channel logo 'ESPN' is visible in the top right corner.`, 0.6542713642120361, 258)\" \n",
              "                        style=\"padding: 10px; border: 1px solid #ddd; border-radius: 5px; background: white; cursor: pointer; text-align: left; transition: all 0.3s;\"\n",
              "                        onmouseover=\"this.style.background='#e3f2fd'; this.style.borderColor='#2196F3';\"\n",
              "                        onmouseout=\"this.style.background='white'; this.style.borderColor='#ddd';\">\n",
              "                    <div style=\"font-weight: bold; color: #1976D2; margin-bottom: 5px;\">\n",
              "                        ⏱️ 505.9s (Frame 258) | Score: 0.654\n",
              "                    </div>\n",
              "                    <div style=\"font-size: 12px; color: #666;\">\n",
              "                        The image shows a soccer match scene. In the foreground, the...\n",
              "                    </div>\n",
              "                </button>\n",
              "\n",
              "                <button onclick=\"jumpToScene(696.9, `The video frame shows a soccer match in progress. A referee wearing a bright yellow shirt and black shorts is walking on the field. The referee's shirt has several sponsor logos, including 'Würth' and 'Hankook.' The playing field is green grass. \n",
              "\n",
              "On the top left of the frame, the scoreboard indicates 'RMA 1-0 FCB,' showing that RMA is leading FCB. Next to the scoreboard, the time displayed is '07:07.' In the top right corner, the broadcast logos 'ESPN' and 'ABC Live' are visible. \n",
              "\n",
              "Another person, likely a player, is partially visible in the lower part of the frame.`, 0.6579287052154541, 355)\" \n",
              "                        style=\"padding: 10px; border: 1px solid #ddd; border-radius: 5px; background: white; cursor: pointer; text-align: left; transition: all 0.3s;\"\n",
              "                        onmouseover=\"this.style.background='#e3f2fd'; this.style.borderColor='#2196F3';\"\n",
              "                        onmouseout=\"this.style.background='white'; this.style.borderColor='#ddd';\">\n",
              "                    <div style=\"font-weight: bold; color: #1976D2; margin-bottom: 5px;\">\n",
              "                        ⏱️ 696.9s (Frame 355) | Score: 0.658\n",
              "                    </div>\n",
              "                    <div style=\"font-size: 12px; color: #666;\">\n",
              "                        The video frame shows a soccer match in progress. A referee ...\n",
              "                    </div>\n",
              "                </button>\n",
              "\n",
              "                <button onclick=\"jumpToScene(956.8, `The frame shows a soccer match in progress. A player wearing a pink and navy blue jersey with a crest and a sponsor logo is sitting on the ground. Another player, wearing a white jersey, is partially in the foreground, obscuring some of the sitting player. The scoreboard at the top left corner displays 'RMA 1 - 0 FCB' with 11:27 as the time. The ESPN and ESPN+ LIVE logos are visible in the top right corner. The background is red, suggesting an advertisement board.`, 0.6655800938606262, 487)\" \n",
              "                        style=\"padding: 10px; border: 1px solid #ddd; border-radius: 5px; background: white; cursor: pointer; text-align: left; transition: all 0.3s;\"\n",
              "                        onmouseover=\"this.style.background='#e3f2fd'; this.style.borderColor='#2196F3';\"\n",
              "                        onmouseout=\"this.style.background='white'; this.style.borderColor='#ddd';\">\n",
              "                    <div style=\"font-weight: bold; color: #1976D2; margin-bottom: 5px;\">\n",
              "                        ⏱️ 956.8s (Frame 487) | Score: 0.666\n",
              "                    </div>\n",
              "                    <div style=\"font-size: 12px; color: #666;\">\n",
              "                        The frame shows a soccer match in progress. A player wearing...\n",
              "                    </div>\n",
              "                </button>\n",
              "\n",
              "            </div>\n",
              "        </div>\n",
              "        \n",
              "        <!-- Video Controls Info -->\n",
              "        <div style=\"margin-top: 20px; padding: 15px; background: #fff3cd; border-radius: 8px; border-left: 4px solid #ffc107;\">\n",
              "            <h4 style=\"margin: 0 0 10px 0; color: #856404;\">💡 How to Use</h4>\n",
              "            <ul style=\"margin: 0; color: #856404; font-size: 14px;\">\n",
              "                <li>Click any timestamp button to jump to that scene</li>\n",
              "                <li>Use video controls to play, pause, and adjust volume</li>\n",
              "                <li>Scene descriptions appear above when you select a timestamp</li>\n",
              "            </ul>\n",
              "        </div>\n",
              "    </div>\n",
              "\n",
              "    <script>\n",
              "        function jumpToScene(timestamp, description, score, frameNumber) {\n",
              "            const video = document.getElementById('videoPlayer');\n",
              "            const sceneDesc = document.getElementById('sceneDescription');\n",
              "            \n",
              "            // Jump to timestamp\n",
              "            video.currentTime = timestamp;\n",
              "            \n",
              "            // Update scene description\n",
              "            sceneDesc.innerHTML = `\n",
              "                <div style=\"margin-bottom: 10px;\">\n",
              "                    <strong>🎬 Frame ${frameNumber} at ${timestamp}s (Score: ${score.toFixed(3)})</strong>\n",
              "                </div>\n",
              "                <div style=\"line-height: 1.5;\">\n",
              "                    ${description}\n",
              "                </div>\n",
              "            `;\n",
              "            \n",
              "            // Auto-play if paused\n",
              "            if (video.paused) {\n",
              "                video.play().catch(e => console.log('Auto-play prevented by browser'));\n",
              "            }\n",
              "            \n",
              "            // Scroll video into view\n",
              "            video.scrollIntoView({ behavior: 'smooth', block: 'nearest' });\n",
              "        }\n",
              "        \n",
              "        // Add time update listener to show current time\n",
              "        document.getElementById('videoPlayer').addEventListener('timeupdate', function() {\n",
              "            const currentTime = this.currentTime;\n",
              "            // You could add real-time scene detection here\n",
              "        });\n",
              "        \n",
              "        // Add keyboard shortcuts\n",
              "        document.addEventListener('keydown', function(e) {\n",
              "            const video = document.getElementById('videoPlayer');\n",
              "            \n",
              "            switch(e.key) {\n",
              "                case ' ':  // Spacebar to play/pause\n",
              "                    e.preventDefault();\n",
              "                    if (video.paused) {\n",
              "                        video.play();\n",
              "                    } else {\n",
              "                        video.pause();\n",
              "                    }\n",
              "                    break;\n",
              "                case 'ArrowLeft':  // Left arrow to go back 10s\n",
              "                    e.preventDefault();\n",
              "                    video.currentTime = Math.max(0, video.currentTime - 10);\n",
              "                    break;\n",
              "                case 'ArrowRight':  // Right arrow to go forward 10s\n",
              "                    e.preventDefault();\n",
              "                    video.currentTime = Math.min(video.duration, video.currentTime + 10);\n",
              "                    break;\n",
              "            }\n",
              "        });\n",
              "    </script>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🎬 Video player created with 5 scenes\n",
            "📁 Video: video.mp4\n",
            "⚠️  Large video file - serving from local path\n"
          ]
        }
      ],
      "source": [
        "# Example usage\n",
        "video_path = \"videos/video.mp4\"  # Replace with your actual video path\n",
        "\n",
        "print(\"🎬 Creating interactive video player...\")\n",
        "create_video_player_with_scenes(\n",
        "    video_path,\n",
        "    full_fidelity_results,\n",
        "    user_query=\"Get me a scene where a player is injured or on the ground\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 5: Making Things Agentic"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "! pip install -Uq openai-agents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {},
      "outputs": [],
      "source": [
        "OPENAI_MODEL = \"gpt-4o\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 5.1 Creating tools for our agents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "metadata": {},
      "outputs": [],
      "source": [
        "from agents.tool import function_tool\n",
        "\n",
        "\n",
        "@function_tool\n",
        "def get_frames_from_scene_description(scene_description: str) -> List[str]:\n",
        "    \"\"\"\n",
        "    Get frames from a scene description provided by the user\n",
        "\n",
        "    Args:\n",
        "        scene_description (str): The scene description to search for\n",
        "\n",
        "    Returns:\n",
        "        List[str]: A list of frame numbers that are relevant to the scene\n",
        "    \"\"\"\n",
        "\n",
        "    print(f\"Getting frames from scene description: {scene_description}\")\n",
        "\n",
        "    # Call the hybrid search function\n",
        "    results = hybrid_search(\n",
        "        user_query=scene_description,\n",
        "        collection=db[FRAME_INTELLIGENCE_METADATA],\n",
        "        top_n=5,\n",
        "        vector_search_index_name=\"vector_search_index_scalar\",\n",
        "        text_search_index_name=\"text_search_index\",\n",
        "        vector_weight=0.5,\n",
        "        text_weight=0.5,\n",
        "    )\n",
        "\n",
        "    print(f\"Frames found: {len(results)} for scene description: {scene_description}\")\n",
        "    return results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def get_frames_from_scene_image(scene_image: str) -> List[str]:\n",
        "    \"\"\"\n",
        "    Get frames from a scene image provided by the user\n",
        "\n",
        "    Args:\n",
        "        scene_description (str): The scene description to search for\n",
        "\n",
        "    Returns:\n",
        "        List[str]: A list of frame numbers that are relevant to the scene\n",
        "    \"\"\"\n",
        "\n",
        "    # Call the hybrid search function\n",
        "    results = hybrid_search(\n",
        "        user_query=scene_image,\n",
        "        collection=db[FRAME_INTELLIGENCE_METADATA],\n",
        "        top_n=5,\n",
        "        vector_search_index_name=\"vector_search_index_scalar\",\n",
        "        text_search_index_name=\"text_search_index\",\n",
        "        vector_weight=0.5,\n",
        "        text_weight=0.5,\n",
        "    )\n",
        "    return results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import List\n",
        "\n",
        "from pydantic import BaseModel\n",
        "\n",
        "\n",
        "# Define the FrameData class\n",
        "class FrameData(BaseModel):\n",
        "    frame_description: str\n",
        "    frame_number: int\n",
        "    frame_timestamp: float\n",
        "    score: float\n",
        "    filename: str = \"\"\n",
        "    embedding: List[float] = []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Next agent displays the video player with the frames that are relevant to the scene description\n",
        "\n",
        "\n",
        "@function_tool\n",
        "def show_video_player_with_frames(frames: List[FrameData], user_query: str) -> str:\n",
        "    \"\"\"\n",
        "    Show the video player with the frame results from the frames intelligence metadata collection.\n",
        "\n",
        "    Args:\n",
        "        frames (List[FrameData]): A list of frame data objects that are relevant to the scene\n",
        "        user_query (str): The user query that triggered the intial search and is used to create the video player\n",
        "\n",
        "    Returns:\n",
        "        str: Success message\n",
        "    \"\"\"\n",
        "    # Convert Pydantic models to dictionaries\n",
        "    frames_dict = [frame.model_dump() for frame in frames]\n",
        "\n",
        "    # TODO: this needs to come from the frames intelligence metadata collection\n",
        "    video_path = \"videos/video.mp4\"  # Replace with your actual video path\n",
        "\n",
        "    print(f\"🎬 Creating interactive video player... with query: {user_query}\")\n",
        "    create_video_player_with_scenes(video_path, frames_dict, user_query=user_query)\n",
        "\n",
        "    return f\"Video player created with {len(frames)} relevant scenes for query: '{user_query}'\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {},
      "outputs": [],
      "source": [
        "from agents import Agent, Runner\n",
        "\n",
        "frame_from_description_agent = Agent(\n",
        "    name=\"Frame From Descripton Retrieval Agent \",\n",
        "    instructions=\"You provide detailed information about frames from a video based on a scene description provided by the user. Always cite your sources.\",\n",
        "    handoff_description=\"A frame retrieval specialist that takes in a scene description that is a string provided by the user and returns a list of frames that are relevant to the scene\",\n",
        "    tools=[get_frames_from_scene_description],\n",
        ")\n",
        "\n",
        "frame_from_scene_image_agent = Agent(\n",
        "    name=\"Frame From Scene Image Retrieval Agent\",\n",
        "    instructions=\"You provide detailed information about frames from a video based on a scene image provided by the user. Always cite your sources.\",\n",
        "    handoff_description=\"A frame retrieval specialist that takes in a scene image that is a string provided by the user and returns a list of frames that are relevant to the scene\",\n",
        "    tools=[get_frames_from_scene_image],\n",
        ")\n",
        "\n",
        "show_video_player_agent = Agent(\n",
        "    name=\"Show Video Player Agent\",\n",
        "    instructions=\"You display the video player with the frames that are relevant to the scene description\",\n",
        "    handoff_description=\"A video player specialist that takes in a list of frames that are relevant to the scene and displays them in a video player\",\n",
        "    tools=[show_video_player_with_frames],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create an orchestrator agent that can use both specialized agents as tools\n",
        "orchestrator_agent = Agent(\n",
        "    name=\"video_intelligence_orchestrator\",\n",
        "    instructions=(\n",
        "        \"You are a video intelligence assistant. Your job is to help users by retrieving relevant information using your tools.\\n\\n\"\n",
        "        \"IMPORTANT RULES:\\n\"\n",
        "        \"1. ALWAYS use frame_from_description_agent when a query mentions a scene description\\n\"\n",
        "        \"2. ALWAYS use frame_from_scene_image_agent when a query mentions a scene image\\n\"\n",
        "        \"3. If a query requires BOTH frame from description AND frame from scene image, use BOTH tools in sequence\\n\"\n",
        "        \"4. NEVER attempt to provide video intelligence information without using your tools\\n\"\n",
        "        \"5. Each tool provides different types of information - use all appropriate tools for complete assistance\"\n",
        "    ),\n",
        "    tools=[\n",
        "        frame_from_description_agent.as_tool(\n",
        "            tool_name=\"frame_from_description\",\n",
        "            tool_description=\"Get frames from a scene description\",\n",
        "        ),\n",
        "        frame_from_scene_image_agent.as_tool(\n",
        "            tool_name=\"frame_from_scene_image\",\n",
        "            tool_description=\"Get frames from a scene image\",\n",
        "        ),\n",
        "        show_video_player_agent.as_tool(\n",
        "            tool_name=\"show_video_player\",\n",
        "            tool_description=\"Show the video player with the frames that are relevant to the scene\",\n",
        "        ),\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "metadata": {},
      "outputs": [],
      "source": [
        "from agents import ItemHelpers, MessageOutputItem, trace\n",
        "\n",
        "\n",
        "async def video_intelligence_assistant(user_query):\n",
        "    \"\"\"Run the complete virtual primary care assistant workflow\"\"\"\n",
        "    # First, have the orchestrator determine which tools to use\n",
        "    with trace(\"Orchestrator evaluator\"):\n",
        "        orchestrator_result = await Runner.run(orchestrator_agent, user_query)\n",
        "\n",
        "        # Print intermediate steps for debugging/transparency\n",
        "        print(\"\\n--- Orchestrator Processing Steps ---\")\n",
        "        for item in orchestrator_result.new_items:\n",
        "            if isinstance(item, MessageOutputItem):\n",
        "                text = ItemHelpers.text_message_output(item)\n",
        "                if text:\n",
        "                    print(f\"  - Information gathering step: {text}\")\n",
        "\n",
        "        print(\n",
        "            f\"\\n\\n--- Final Video Intelligence Response ---\\n{orchestrator_result.final_output}\"\n",
        "        )\n",
        "        print()\n",
        "\n",
        "    return orchestrator_result.final_output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "metadata": {},
      "outputs": [],
      "source": [
        "import asyncio\n",
        "\n",
        "import nest_asyncio\n",
        "\n",
        "# Apply nest_asyncio to patch the event loop\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "metadata": {},
      "outputs": [],
      "source": [
        "def run_video_intelligence_assistant(query):\n",
        "    # Create a new event loop\n",
        "    loop = asyncio.new_event_loop()\n",
        "    asyncio.set_event_loop(loop)\n",
        "\n",
        "    # Run the async function and get the result\n",
        "    result = loop.run_until_complete(video_intelligence_assistant(query))\n",
        "\n",
        "    # Clean up\n",
        "    loop.close()\n",
        "\n",
        "    return result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Now call the function this way\n",
        "query = input(\"What video intelligence can I help you with today? \")\n",
        "run_video_intelligence_assistant(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 6: Live Stream Diagonsis"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "metadata": {},
      "outputs": [],
      "source": [
        "import threading\n",
        "import time\n",
        "from typing import Optional\n",
        "\n",
        "import yt_dlp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "metadata": {},
      "outputs": [],
      "source": [
        "def search_similar_incidents(embedding, collection, threshold=0.7):\n",
        "    \"\"\"Search for similar incidents in database\"\"\"\n",
        "    try:\n",
        "        # Simplified vector search\n",
        "        pipeline = [\n",
        "            {\n",
        "                \"$vectorSearch\": {\n",
        "                    \"index\": \"incident_vector_index_scalar\",\n",
        "                    \"path\": \"embedding\",\n",
        "                    \"queryVector\": embedding,\n",
        "                    \"numCandidates\": 100,\n",
        "                    \"limit\": 10,\n",
        "                }\n",
        "            },\n",
        "            {\"$addFields\": {\"similarity_score\": {\"$meta\": \"vectorSearchScore\"}}},\n",
        "            {\"$match\": {\"similarity_score\": {\"$gte\": threshold}}},\n",
        "        ]\n",
        "\n",
        "        results = list(collection.aggregate(pipeline))\n",
        "        return results\n",
        "    except Exception as e:\n",
        "        print(f\"Error searching incidents: {e}\")\n",
        "        return []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Global variables for the monitoring system\n",
        "current_monitoring = {\n",
        "    \"active\": False,\n",
        "    \"cap\": None,\n",
        "    \"frame_count\": 0,\n",
        "    \"incidents\": [],\n",
        "    \"stats\": {\"frames_processed\": 0, \"incidents_detected\": 0},\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {},
      "outputs": [],
      "source": [
        "# =============================================================================\n",
        "# HELPER FUNCTIONS\n",
        "# =============================================================================\n",
        "from datetime import datetime\n",
        "\n",
        "\n",
        "def _monitoring_worker():\n",
        "    \"\"\"Background worker for processing frames\"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    # Connect to MongoDB\n",
        "    try:\n",
        "        collection = db[PREVIOUS_FRAME_INCIDENTS]\n",
        "    except Exception as e:\n",
        "        print(f\"Warning: Could not connect to MongoDB: {e}\")\n",
        "        collection = None\n",
        "\n",
        "    while current_monitoring[\"active\"]:\n",
        "        try:\n",
        "            if not current_monitoring[\"cap\"]:\n",
        "                break\n",
        "\n",
        "            ret, frame = current_monitoring[\"cap\"].read()\n",
        "            if not ret:\n",
        "                break\n",
        "\n",
        "            current_monitoring[\"frame_count\"] += 1\n",
        "\n",
        "            # Process every N frames\n",
        "            if (\n",
        "                current_monitoring[\"frame_count\"]\n",
        "                % current_monitoring.get(\"check_interval\", 30)\n",
        "                == 0\n",
        "            ):\n",
        "                _process_frame_for_incidents(frame, collection)\n",
        "\n",
        "            time.sleep(0.01)  # Small delay\n",
        "\n",
        "        except Exception as e:\n",
        "            print(f\"Monitoring error: {e}\")\n",
        "            break\n",
        "\n",
        "\n",
        "def _process_frame_for_incidents(frame, collection):\n",
        "    \"\"\"Process a single frame for incident detection\"\"\"\n",
        "    try:\n",
        "        # Convert frame to embedding\n",
        "        rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
        "        pil_image = Image.fromarray(rgb_frame)\n",
        "        embedding = get_voyage_embedding(pil_image, \"query\")\n",
        "\n",
        "        # Search for similar incidents\n",
        "        if collection is not None:\n",
        "            incidents = search_similar_incidents(embedding, collection)\n",
        "\n",
        "            for incident in incidents:\n",
        "                incident_data = {\n",
        "                    \"incident_type\": incident.get(\"incident_type\", \"Unknown\"),\n",
        "                    \"network_issue\": incident.get(\"network_issue\", \"Unknown\"),\n",
        "                    \"resolution_text\": incident.get(\"resolution_text\", \"No solution\"),\n",
        "                    \"similarity_score\": incident.get(\"similarity_score\", 0),\n",
        "                    \"frame_number\": current_monitoring[\"frame_count\"],\n",
        "                    \"timestamp\": datetime.now().isoformat(),\n",
        "                }\n",
        "\n",
        "                current_monitoring[\"incidents\"].append(incident_data)\n",
        "                current_monitoring[\"stats\"][\"incidents_detected\"] += 1\n",
        "\n",
        "        current_monitoring[\"stats\"][\"frames_processed\"] += 1\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Frame processing error: {e}\")\n",
        "\n",
        "\n",
        "def _add_incident_overlays(frame):\n",
        "    \"\"\"Add incident detection overlays to frame\"\"\"\n",
        "    # Get recent incidents (last 10 seconds worth)\n",
        "    recent_incidents = current_monitoring[\"incidents\"][-3:]  # Show last 3\n",
        "\n",
        "    if not recent_incidents:\n",
        "        return frame\n",
        "\n",
        "    # Add overlay background\n",
        "    overlay_y = 30\n",
        "    for incident in recent_incidents:\n",
        "        # Create text\n",
        "        text_lines = [\n",
        "            f\"⚠️ {incident['incident_type']} (Score: {incident['similarity_score']:.2f})\",\n",
        "            f\"Issue: {incident['network_issue'][:40]}...\",\n",
        "            f\"Solution: {incident['resolution_text'][:40]}...\",\n",
        "        ]\n",
        "\n",
        "        # Draw background\n",
        "        max_width = max(len(line) for line in text_lines) * 8\n",
        "        rect_height = len(text_lines) * 20 + 10\n",
        "\n",
        "        cv2.rectangle(\n",
        "            frame, (10, overlay_y), (max_width, overlay_y + rect_height), (0, 0, 0), -1\n",
        "        )\n",
        "        cv2.rectangle(\n",
        "            frame,\n",
        "            (10, overlay_y),\n",
        "            (max_width, overlay_y + rect_height),\n",
        "            (0, 255, 255),\n",
        "            2,\n",
        "        )\n",
        "\n",
        "        # Draw text\n",
        "        for i, line in enumerate(text_lines):\n",
        "            y_pos = overlay_y + 15 + (i * 20)\n",
        "            cv2.putText(\n",
        "                frame,\n",
        "                line,\n",
        "                (15, y_pos),\n",
        "                cv2.FONT_HERSHEY_SIMPLEX,\n",
        "                0.5,\n",
        "                (255, 255, 255),\n",
        "                1,\n",
        "            )\n",
        "\n",
        "        overlay_y += rect_height + 10\n",
        "\n",
        "    # Add stats\n",
        "    stats_text = f\"Frames: {current_monitoring['stats']['frames_processed']} | Incidents: {current_monitoring['stats']['incidents_detected']}\"\n",
        "    cv2.putText(\n",
        "        frame,\n",
        "        stats_text,\n",
        "        (10, frame.shape[0] - 20),\n",
        "        cv2.FONT_HERSHEY_SIMPLEX,\n",
        "        0.5,\n",
        "        (0, 255, 0),\n",
        "        1,\n",
        "    )\n",
        "\n",
        "    return frame\n",
        "\n",
        "\n",
        "def _print_stats():\n",
        "    \"\"\"Print current statistics\"\"\"\n",
        "    stats = current_monitoring[\"stats\"]\n",
        "    print(\n",
        "        f\"\\n📊 Stats: Frames: {stats['frames_processed']} | Incidents: {stats['incidents_detected']}\"\n",
        "    )\n",
        "    print(f\"Current frame: {current_monitoring['frame_count']}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {},
      "outputs": [],
      "source": [
        "def extract_youtube_url(youtube_url: str) -> Optional[str]:\n",
        "    \"\"\"Extract direct stream URL from YouTube\"\"\"\n",
        "    try:\n",
        "        print(f\"🔍 Extracting stream from: {youtube_url}\")\n",
        "\n",
        "        ydl_opts = {\n",
        "            \"format\": \"best[height<=720]/best\",  # Fallback to any quality\n",
        "            \"quiet\": True,\n",
        "            \"no_warnings\": True,\n",
        "            \"extract_flat\": False,\n",
        "        }\n",
        "\n",
        "        with yt_dlp.YoutubeDL(ydl_opts) as ydl:\n",
        "            print(\"📥 Downloading video info...\")\n",
        "            info = ydl.extract_info(youtube_url, download=False)\n",
        "\n",
        "            if not info:\n",
        "                print(\"❌ No video information extracted\")\n",
        "                return None\n",
        "\n",
        "            print(f\"📺 Video title: {info.get('title', 'Unknown')}\")\n",
        "            print(f\"📺 Uploader: {info.get('uploader', 'Unknown')}\")\n",
        "            print(f\"🔴 Is live: {info.get('is_live', False)}\")\n",
        "            print(f\"⏱️ Duration: {info.get('duration', 'Unknown')}\")\n",
        "\n",
        "            # Get the best format URL\n",
        "            formats = info.get(\"formats\", [])\n",
        "            print(f\"📊 Found {len(formats)} formats\")\n",
        "\n",
        "            if not formats:\n",
        "                print(\"❌ No video formats available\")\n",
        "                return None\n",
        "\n",
        "            # Try to find a good format\n",
        "            best_format = None\n",
        "            for fmt in formats:\n",
        "                if fmt.get(\"vcodec\") != \"none\" and fmt.get(\"url\"):\n",
        "                    if not best_format or (\n",
        "                        fmt.get(\"height\", 0) > best_format.get(\"height\", 0)\n",
        "                    ):\n",
        "                        best_format = fmt\n",
        "                        print(\n",
        "                            f\"📊 Found format: {fmt.get('format_id')} - {fmt.get('width')}x{fmt.get('height')}\"\n",
        "                        )\n",
        "\n",
        "            if not best_format:\n",
        "                print(\"❌ No suitable video format found\")\n",
        "                return None\n",
        "\n",
        "            stream_url = best_format[\"url\"]\n",
        "            print(f\"✅ Stream URL extracted: {stream_url[:60]}...\")\n",
        "            return stream_url\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"💥 YouTube extraction error: {type(e).__name__}: {e}\")\n",
        "\n",
        "        # Common YouTube errors and solutions\n",
        "        if \"Video unavailable\" in str(e):\n",
        "            print(\"💡 Video may be private, deleted, or geo-blocked\")\n",
        "        elif \"Sign in to confirm your age\" in str(e):\n",
        "            print(\"💡 Video requires age verification\")\n",
        "        elif \"This live event has ended\" in str(e):\n",
        "            print(\"💡 Livestream has ended\")\n",
        "        else:\n",
        "            print(\"💡 Try a different YouTube video or check if the URL is correct\")\n",
        "\n",
        "        return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def stop_video_monitoring() -> str:\n",
        "    \"\"\"\n",
        "    Stop current video monitoring.\n",
        "\n",
        "    Returns:\n",
        "        Status message with final statistics\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    if not current_monitoring[\"active\"]:\n",
        "        return \"No active monitoring to stop\"\n",
        "\n",
        "    # Stop monitoring\n",
        "    current_monitoring[\"active\"] = False\n",
        "\n",
        "    if current_monitoring[\"cap\"]:\n",
        "        current_monitoring[\"cap\"].release()\n",
        "\n",
        "    # Get final stats\n",
        "    final_stats = {\n",
        "        \"total_frames\": current_monitoring[\"frame_count\"],\n",
        "        \"frames_processed\": current_monitoring[\"stats\"][\"frames_processed\"],\n",
        "        \"incidents_detected\": current_monitoring[\"stats\"][\"incidents_detected\"],\n",
        "    }\n",
        "\n",
        "    cv2.destroyAllWindows()\n",
        "\n",
        "    return f\"🛑 Monitoring stopped. Final stats: {final_stats}\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def start_video_monitoring(source: str = \"0\", check_interval: int = 30) -> str:\n",
        "    \"\"\"\n",
        "    Start real-time video monitoring for incident detection.\n",
        "\n",
        "    Args:\n",
        "        source: Video source (0 for webcam, file path, or YouTube URL)\n",
        "        check_interval: Check every N frames\n",
        "\n",
        "    Returns:\n",
        "        Status message\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    try:\n",
        "        # Stop existing monitoring\n",
        "        if current_monitoring[\"active\"]:\n",
        "            stop_video_monitoring()\n",
        "\n",
        "        print(f\"🔍 Processing source: {source}\")\n",
        "\n",
        "        # Handle different source types\n",
        "        if source.startswith(\"http\") and \"youtube\" in source:\n",
        "            print(\"📺 Detected YouTube URL, extracting stream...\")\n",
        "            stream_url = extract_youtube_url(source)\n",
        "            if not stream_url:\n",
        "                return f\"❌ Failed to extract YouTube stream URL from: {source}. Video may be private, restricted, or unavailable.\"\n",
        "            print(f\"✅ Extracted stream URL: {stream_url[:60]}...\")\n",
        "            video_source = stream_url\n",
        "        elif source.isdigit():\n",
        "            video_source = int(source)  # Webcam\n",
        "            print(f\"📷 Using webcam: {video_source}\")\n",
        "        else:\n",
        "            video_source = source  # File path\n",
        "            print(f\"📁 Using file: {video_source}\")\n",
        "\n",
        "        # Start video capture with detailed error checking\n",
        "        print(\"🎥 Initializing video capture...\")\n",
        "        cap = cv2.VideoCapture(video_source)\n",
        "\n",
        "        if not cap.isOpened():\n",
        "            error_msg = f\"❌ Could not open video source: {source}\"\n",
        "            if isinstance(video_source, str) and video_source.startswith(\"http\"):\n",
        "                error_msg += \"\\n   - YouTube stream URL may be invalid or expired\"\n",
        "                error_msg += (\n",
        "                    \"\\n   - Try a different video or check if the stream is live\"\n",
        "                )\n",
        "            print(error_msg)\n",
        "            return error_msg\n",
        "\n",
        "        # Test if we can actually read a frame\n",
        "        print(\"🧪 Testing frame capture...\")\n",
        "        ret, test_frame = cap.read()\n",
        "        if not ret or test_frame is None:\n",
        "            cap.release()\n",
        "            error_msg = f\"❌ Could not read frames from video source: {source}\"\n",
        "            if isinstance(video_source, str) and video_source.startswith(\"http\"):\n",
        "                error_msg += \"\\n   - Stream may have ended or be unavailable\"\n",
        "            print(error_msg)\n",
        "            return error_msg\n",
        "\n",
        "        print(f\"✅ Successfully captured test frame: {test_frame.shape}\")\n",
        "\n",
        "        # Initialize monitoring\n",
        "        current_monitoring[\"active\"] = True\n",
        "        current_monitoring[\"cap\"] = cap\n",
        "        current_monitoring[\"frame_count\"] = 0\n",
        "        current_monitoring[\"incidents\"] = []\n",
        "        current_monitoring[\"check_interval\"] = check_interval\n",
        "        current_monitoring[\"source\"] = source\n",
        "\n",
        "        # Start monitoring thread\n",
        "        monitor_thread = threading.Thread(target=_monitoring_worker, daemon=True)\n",
        "        monitor_thread.start()\n",
        "\n",
        "        success_msg = f\"✅ Started monitoring source: {source} (checking every {check_interval} frames)\"\n",
        "        print(success_msg)\n",
        "        return success_msg\n",
        "\n",
        "    except Exception as e:\n",
        "        error_msg = f\"❌ Error starting monitoring: {e}\"\n",
        "        print(f\"💥 Exception details: {type(e).__name__}: {e}\")\n",
        "        return error_msg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {},
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "\n",
        "@function_tool\n",
        "def get_monitoring_status() -> dict:\n",
        "    \"\"\"\n",
        "    Get current monitoring status and statistics.\n",
        "\n",
        "    Returns:\n",
        "        Dictionary with monitoring status and stats\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    return {\n",
        "        \"active\": current_monitoring[\"active\"],\n",
        "        \"frame_count\": current_monitoring[\"frame_count\"],\n",
        "        \"frames_processed\": current_monitoring[\"stats\"][\"frames_processed\"],\n",
        "        \"incidents_detected\": current_monitoring[\"stats\"][\"incidents_detected\"],\n",
        "        \"recent_incidents\": len(current_monitoring[\"incidents\"]),\n",
        "        \"timestamp\": datetime.now().isoformat(),\n",
        "    }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def get_recent_incidents(max_incidents: int = 5) -> list:\n",
        "    \"\"\"\n",
        "    Get recent incident detections.\n",
        "\n",
        "    Args:\n",
        "        max_incidents: Maximum number of incidents to return\n",
        "\n",
        "    Returns:\n",
        "        List of recent incidents\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    recent = current_monitoring[\"incidents\"][-max_incidents:]\n",
        "\n",
        "    # Convert to simple format\n",
        "    incidents = []\n",
        "    for incident in recent:\n",
        "        incidents.append(\n",
        "            {\n",
        "                \"incident_type\": incident.get(\"incident_type\", \"Unknown\"),\n",
        "                \"network_issue\": incident.get(\"network_issue\", \"Unknown issue\"),\n",
        "                \"resolution_text\": incident.get(\n",
        "                    \"resolution_text\", \"No solution available\"\n",
        "                ),\n",
        "                \"similarity_score\": incident.get(\"similarity_score\", 0),\n",
        "                \"frame_number\": incident.get(\"frame_number\", 0),\n",
        "                \"timestamp\": incident.get(\"timestamp\", datetime.now().isoformat()),\n",
        "            }\n",
        "        )\n",
        "\n",
        "    return incidents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def show_video_feed(show_overlays: bool = True) -> str:\n",
        "    \"\"\"\n",
        "    Display the video feed with optional incident overlays.\n",
        "\n",
        "    Args:\n",
        "        show_overlays: Whether to show incident detection overlays\n",
        "\n",
        "    Returns:\n",
        "        Status message\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    if not current_monitoring[\"active\"]:\n",
        "        return \"❌ No active video monitoring to display. Start monitoring first with 'Start monitoring [source]'\"\n",
        "\n",
        "    if not current_monitoring[\"cap\"]:\n",
        "        return \"❌ Video capture not available. There may be an issue with the video source.\"\n",
        "\n",
        "    try:\n",
        "        print(\"🎥 Displaying video feed... (Press 'q' to quit, 's' for stats)\")\n",
        "        print(f\"📺 Source: {current_monitoring.get('source', 'Unknown')}\")\n",
        "        print(f\"🔄 Frame count: {current_monitoring['frame_count']}\")\n",
        "\n",
        "        display_count = 0\n",
        "        max_empty_frames = 10\n",
        "        empty_frame_count = 0\n",
        "\n",
        "        while current_monitoring[\"active\"]:\n",
        "            ret, frame = current_monitoring[\"cap\"].read()\n",
        "            if not ret:\n",
        "                empty_frame_count += 1\n",
        "                print(f\"⚠️ Failed to read frame {empty_frame_count}/{max_empty_frames}\")\n",
        "\n",
        "                if empty_frame_count >= max_empty_frames:\n",
        "                    print(\"❌ Too many failed frame reads, stopping display\")\n",
        "                    break\n",
        "\n",
        "                time.sleep(0.1)  # Wait a bit before trying again\n",
        "                continue\n",
        "\n",
        "            # Reset empty frame counter on successful read\n",
        "            empty_frame_count = 0\n",
        "            display_count += 1\n",
        "\n",
        "            # Add overlays if requested\n",
        "            if show_overlays:\n",
        "                frame = _add_incident_overlays(frame)\n",
        "\n",
        "            # Add display info\n",
        "            info_text = f\"Display: {display_count} | Source: {current_monitoring.get('source', 'Unknown')[:30]}\"\n",
        "            cv2.putText(\n",
        "                frame,\n",
        "                info_text,\n",
        "                (10, frame.shape[0] - 70),\n",
        "                cv2.FONT_HERSHEY_SIMPLEX,\n",
        "                0.5,\n",
        "                (255, 255, 255),\n",
        "                1,\n",
        "            )\n",
        "\n",
        "            # Display frame\n",
        "            cv2.imshow(\"Real-time Video Analysis\", frame)\n",
        "\n",
        "            # Handle key presses\n",
        "            key = cv2.waitKey(1) & 0xFF\n",
        "            if key == ord(\"q\"):\n",
        "                print(\"🛑 Display stopped by user\")\n",
        "                break\n",
        "            elif key == ord(\"s\"):\n",
        "                _print_stats()\n",
        "\n",
        "        cv2.destroyAllWindows()\n",
        "        return f\"✅ Video display closed after showing {display_count} frames\"\n",
        "\n",
        "    except Exception as e:\n",
        "        cv2.destroyAllWindows()\n",
        "        error_msg = f\"❌ Error displaying video: {e}\"\n",
        "        print(f\"💥 Display error: {type(e).__name__}: {e}\")\n",
        "        return error_msg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def show_latest_frame() -> str:\n",
        "    \"\"\"\n",
        "    Capture and display the current frame in Jupyter notebook.\n",
        "\n",
        "    Returns:\n",
        "        Status message with frame info\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    if not current_monitoring[\"active\"] or not current_monitoring[\"cap\"]:\n",
        "        return \"❌ No active video monitoring to capture frame from\"\n",
        "\n",
        "    try:\n",
        "        # Capture current frame\n",
        "        ret, frame = current_monitoring[\"cap\"].read()\n",
        "        if not ret:\n",
        "            return \"❌ Could not capture current frame\"\n",
        "\n",
        "        # Add overlays\n",
        "        frame = _add_incident_overlays(frame)\n",
        "\n",
        "        # Add info\n",
        "        info_text = f\"Frame: {current_monitoring['frame_count']} | Live capture\"\n",
        "        cv2.putText(\n",
        "            frame,\n",
        "            info_text,\n",
        "            (10, 30),\n",
        "            cv2.FONT_HERSHEY_SIMPLEX,\n",
        "            0.7,\n",
        "            (255, 255, 255),\n",
        "            2,\n",
        "        )\n",
        "\n",
        "        # Save current frame\n",
        "        import os\n",
        "\n",
        "        os.makedirs(\"video_frames\", exist_ok=True)\n",
        "        filename = \"video_frames/current_frame.jpg\"\n",
        "        cv2.imwrite(filename, frame)\n",
        "\n",
        "        # Display in notebook using IPython\n",
        "        try:\n",
        "            from IPython.display import Image, display\n",
        "\n",
        "            display(Image(filename))\n",
        "            return f\"✅ Current frame displayed (Frame #{current_monitoring['frame_count']})\"\n",
        "        except ImportError:\n",
        "            return f\"✅ Current frame saved to {filename} (Frame #{current_monitoring['frame_count']})\"\n",
        "\n",
        "    except Exception as e:\n",
        "        return f\"❌ Error capturing frame: {e}\"\n",
        "\n",
        "\n",
        "@function_tool\n",
        "def create_video_summary() -> str:\n",
        "    \"\"\"\n",
        "    Create a summary of the current monitoring session.\n",
        "\n",
        "    Returns:\n",
        "        Summary of monitoring statistics and recent activity\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    if not current_monitoring.get(\"active\", False):\n",
        "        return \"❌ No active monitoring session\"\n",
        "\n",
        "    try:\n",
        "        summary = f\"\"\"\n",
        "📊 **Video Monitoring Summary**\n",
        "\n",
        "🎬 **Source**: {current_monitoring.get('source', 'Unknown')}\n",
        "📊 **Status**: {'🟢 Active' if current_monitoring['active'] else '🔴 Stopped'}\n",
        "🔢 **Total Frames**: {current_monitoring.get('frame_count', 0)}\n",
        "⚙️ **Processed**: {current_monitoring.get('stats', {}).get('frames_processed', 0)}\n",
        "🚨 **Incidents**: {current_monitoring.get('stats', {}).get('incidents_detected', 0)}\n",
        "⏱️ **Check Interval**: Every {current_monitoring.get('check_interval', 'Unknown')} frames\n",
        "\n",
        "🎯 **Recent Incidents**: {len(current_monitoring.get('incidents', []))} detected\n",
        "        \"\"\"\n",
        "\n",
        "        # Add recent incidents if any\n",
        "        recent_incidents = current_monitoring.get(\"incidents\", [])[-3:]  # Last 3\n",
        "        if recent_incidents:\n",
        "            summary += \"\\n🚨 **Latest Incidents**:\\n\"\n",
        "            for i, incident in enumerate(recent_incidents, 1):\n",
        "                summary += f\"   {i}. {incident.get('incident_type', 'Unknown')} (Score: {incident.get('similarity_score', 0):.2f})\\n\"\n",
        "        else:\n",
        "            summary += \"\\n✅ **No Recent Incidents Detected**\"\n",
        "\n",
        "        return summary\n",
        "\n",
        "    except Exception as e:\n",
        "        return f\"❌ Error creating summary: {e}\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def get_youtube_stream_info(youtube_url: str) -> dict:\n",
        "    \"\"\"\n",
        "    Get direct stream information from a YouTube URL.\n",
        "\n",
        "    Args:\n",
        "        youtube_url: YouTube video/livestream URL\n",
        "\n",
        "    Returns:\n",
        "        Dictionary with stream information and direct URL\n",
        "    \"\"\"\n",
        "    try:\n",
        "        ydl_opts = {\n",
        "            \"format\": \"best[height<=720]\",\n",
        "            \"quiet\": True,\n",
        "            \"no_warnings\": True,\n",
        "            \"extractaudio\": False,\n",
        "        }\n",
        "\n",
        "        with yt_dlp.YoutubeDL(ydl_opts) as ydl:\n",
        "            info = ydl.extract_info(youtube_url, download=False)\n",
        "\n",
        "            if not info:\n",
        "                return {\"error\": \"Could not extract video information\"}\n",
        "\n",
        "            # Get the best format URL\n",
        "            formats = info.get(\"formats\", [])\n",
        "            best_format = None\n",
        "\n",
        "            for fmt in formats:\n",
        "                if fmt.get(\"vcodec\") != \"none\" and fmt.get(\"url\"):\n",
        "                    if not best_format or (\n",
        "                        fmt.get(\"height\", 0) > best_format.get(\"height\", 0)\n",
        "                    ):\n",
        "                        best_format = fmt\n",
        "\n",
        "            if not best_format:\n",
        "                return {\"error\": \"No suitable video format found\"}\n",
        "\n",
        "            return {\n",
        "                \"success\": True,\n",
        "                \"title\": info.get(\"title\", \"Unknown\"),\n",
        "                \"uploader\": info.get(\"uploader\", \"Unknown\"),\n",
        "                \"is_live\": info.get(\"is_live\", False),\n",
        "                \"duration\": info.get(\"duration\"),\n",
        "                \"direct_url\": best_format[\"url\"],\n",
        "                \"width\": best_format.get(\"width\"),\n",
        "                \"height\": best_format.get(\"height\"),\n",
        "                \"fps\": best_format.get(\"fps\"),\n",
        "                \"original_url\": youtube_url,\n",
        "            }\n",
        "\n",
        "    except Exception as e:\n",
        "        return {\"error\": f\"Failed to extract stream: {e}\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {},
      "outputs": [],
      "source": [
        "@function_tool\n",
        "def show_video_feed_jupyter(\n",
        "    show_overlays: bool = True, save_frames: bool = True\n",
        ") -> str:\n",
        "    \"\"\"\n",
        "    Display video feed in Jupyter notebook (saves frames instead of using cv2.imshow).\n",
        "\n",
        "    Args:\n",
        "        show_overlays: Whether to show incident detection overlays\n",
        "        save_frames: Whether to save frames to files\n",
        "\n",
        "    Returns:\n",
        "        Status message\n",
        "    \"\"\"\n",
        "    global current_monitoring\n",
        "\n",
        "    if not current_monitoring[\"active\"]:\n",
        "        return \"❌ No active video monitoring to display. Start monitoring first.\"\n",
        "\n",
        "    if not current_monitoring[\"cap\"]:\n",
        "        return \"❌ Video capture not available.\"\n",
        "\n",
        "    try:\n",
        "        print(\"🎥 Jupyter-friendly video display starting...\")\n",
        "        print(f\"📺 Source: {current_monitoring.get('source', 'Unknown')}\")\n",
        "        print(f\"🔄 Current frame count: {current_monitoring['frame_count']}\")\n",
        "        print(\"📁 Frames will be saved to 'video_frames/' directory\")\n",
        "        print(\"🛑 Run 'Stop monitoring' to stop\")\n",
        "\n",
        "        # Create frames directory\n",
        "        import os\n",
        "\n",
        "        os.makedirs(\"video_frames\", exist_ok=True)\n",
        "\n",
        "        frame_save_count = 0\n",
        "        max_frames_to_save = 10  # Save every 30th frame for demo\n",
        "\n",
        "        for i in range(max_frames_to_save):\n",
        "            if not current_monitoring[\"active\"]:\n",
        "                break\n",
        "\n",
        "            ret, frame = current_monitoring[\"cap\"].read()\n",
        "            if not ret:\n",
        "                print(f\"⚠️ Could not read frame {i+1}\")\n",
        "                continue\n",
        "\n",
        "            # Add overlays if requested\n",
        "            if show_overlays:\n",
        "                frame = _add_incident_overlays(frame)\n",
        "\n",
        "            # Add frame info\n",
        "            info_text = f\"Frame: {current_monitoring['frame_count']} | {current_monitoring.get('source', 'Unknown')[:30]}\"\n",
        "            cv2.putText(\n",
        "                frame,\n",
        "                info_text,\n",
        "                (10, 30),\n",
        "                cv2.FONT_HERSHEY_SIMPLEX,\n",
        "                0.7,\n",
        "                (255, 255, 255),\n",
        "                2,\n",
        "            )\n",
        "\n",
        "            # Save frame\n",
        "            if save_frames:\n",
        "                filename = f\"video_frames/frame_{frame_save_count:04d}.jpg\"\n",
        "                cv2.imwrite(filename, frame)\n",
        "                frame_save_count += 1\n",
        "                print(f\"💾 Saved frame {frame_save_count}: {filename}\")\n",
        "\n",
        "            # Small delay\n",
        "            import time\n",
        "\n",
        "            time.sleep(0.5)  # Save a frame every 0.5 seconds\n",
        "\n",
        "        return f\"✅ Saved {frame_save_count} frames to 'video_frames/' directory. Monitoring continues in background.\"\n",
        "\n",
        "    except Exception as e:\n",
        "        error_msg = f\"❌ Error in Jupyter display: {e}\"\n",
        "        print(f\"💥 Display error: {type(e).__name__}: {e}\")\n",
        "        return error_msg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {},
      "outputs": [],
      "source": [
        "# =============================================================================\n",
        "# AGENTS\n",
        "# =============================================================================\n",
        "\n",
        "from agents import Agent, Runner\n",
        "\n",
        "# Simple monitoring agent\n",
        "monitoring_agent = Agent(\n",
        "    name=\"Video Monitoring Agent\",\n",
        "    instructions=\"You start and stop video monitoring for incident detection. You can handle webcam, files, and YouTube streams.\",\n",
        "    handoff_description=\"Specialist for starting and managing video stream monitoring\",\n",
        "    tools=[start_video_monitoring, stop_video_monitoring, get_monitoring_status],\n",
        ")\n",
        "\n",
        "# Incident analysis agent\n",
        "incident_agent = Agent(\n",
        "    name=\"Incident Analysis Agent\",\n",
        "    instructions=\"You analyze detected incidents and provide detailed reports about network issues and solutions.\",\n",
        "    handoff_description=\"Specialist for analyzing and reporting video incidents\",\n",
        "    tools=[get_recent_incidents, get_monitoring_status],\n",
        ")\n",
        "\n",
        "# Video display agent\n",
        "display_agent = Agent(\n",
        "    name=\"Video Display Agent\",\n",
        "    instructions=\"You show the live video feed with incident detection overlays.\",\n",
        "    handoff_description=\"Specialist for displaying video feeds with overlays\",\n",
        "    tools=[\n",
        "        show_video_feed_jupyter,\n",
        "        get_monitoring_status,\n",
        "        show_latest_frame,\n",
        "        create_video_summary,\n",
        "    ],\n",
        ")\n",
        "\n",
        "# YouTube search agent\n",
        "youtube_agent = Agent(\n",
        "    name=\"YouTube Stream Agent\",\n",
        "    instructions=\"You search for YouTube livestreams and help set up monitoring.\",\n",
        "    handoff_description=\"Specialist for finding and setting up YouTube livestream monitoring\",\n",
        "    tools=[get_youtube_stream_info, start_video_monitoring],\n",
        ")\n",
        "\n",
        "# Main orchestrator\n",
        "video_orchestrator = Agent(\n",
        "    name=\"Real-time Video Orchestrator\",\n",
        "    instructions=(\n",
        "        \"You coordinate real-time video monitoring and incident detection. \"\n",
        "        \"Use monitoring_agent to start/stop monitoring, incident_agent for analysis, \"\n",
        "        \"display_agent to show video feeds, and youtube_agent for YouTube streams. \"\n",
        "        \"Always provide clear status updates and help users through the complete workflow.\"\n",
        "    ),\n",
        "    tools=[\n",
        "        monitoring_agent.as_tool(\n",
        "            tool_name=\"monitoring\",\n",
        "            tool_description=\"Start, stop, and manage video monitoring\",\n",
        "        ),\n",
        "        incident_agent.as_tool(\n",
        "            tool_name=\"incidents\",\n",
        "            tool_description=\"Analyze and report detected incidents\",\n",
        "        ),\n",
        "        display_agent.as_tool(\n",
        "            tool_name=\"display\",\n",
        "            tool_description=\"Show video feed with incident overlays\",\n",
        "        ),\n",
        "        youtube_agent.as_tool(\n",
        "            tool_name=\"youtube\",\n",
        "            tool_description=\"Search and monitor YouTube livestreams\",\n",
        "        ),\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {},
      "outputs": [],
      "source": [
        "async def quick_start_youtube(youtube_url):\n",
        "    \"\"\"Quick start with YouTube monitoring\"\"\"\n",
        "    print(f\"🚀 Starting YouTube monitoring: {youtube_url}\")\n",
        "\n",
        "    # First test the URL\n",
        "    test_result = await Runner.run(\n",
        "        video_orchestrator, f\"Test this YouTube URL: {youtube_url}\"\n",
        "    )\n",
        "    print(\"Test result:\", test_result.final_output)\n",
        "\n",
        "    # If test passed, try monitoring\n",
        "    if \"test PASSED\" in test_result.final_output:\n",
        "        result = await Runner.run(\n",
        "            video_orchestrator, f\"Start monitoring this YouTube stream: {youtube_url}\"\n",
        "        )\n",
        "        print(\"Response:\", result.final_output)\n",
        "\n",
        "        print(\"\\n🎥 Showing video feed...\")\n",
        "        display_result = await Runner.run(\n",
        "            video_orchestrator, \"Show the video feed with incident overlays\"\n",
        "        )\n",
        "        print(\"Display response:\", display_result.final_output)\n",
        "\n",
        "        return result\n",
        "    else:\n",
        "        print(\"❌ URL test failed, not starting monitoring\")\n",
        "        return test_result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {},
      "outputs": [],
      "source": [
        "async def get_youtube_info_example(youtube_url):\n",
        "    \"\"\"Example: Get YouTube stream information\"\"\"\n",
        "    print(f\"🔍 Getting stream info for: {youtube_url}\")\n",
        "\n",
        "    result = await Runner.run(\n",
        "        video_orchestrator, f\"Get stream info for this YouTube URL: {youtube_url}\"\n",
        "    )\n",
        "    print(\"Response:\", result.final_output)\n",
        "\n",
        "    return result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {},
      "outputs": [],
      "source": [
        "async def video_monitoring_assistant(user_query):\n",
        "    \"\"\"Main video monitoring assistant function matching your pattern\"\"\"\n",
        "    try:\n",
        "        from agents import ItemHelpers, MessageOutputItem, trace\n",
        "    except ImportError:\n",
        "        # Fallback if agents utilities aren't available\n",
        "        result = await Runner.run(video_orchestrator, user_query)\n",
        "        return result.final_output\n",
        "\n",
        "    # First, have the orchestrator determine which tools to use\n",
        "    with trace(\"Video monitoring orchestrator\"):\n",
        "        orchestrator_result = await Runner.run(video_orchestrator, user_query)\n",
        "\n",
        "        # Print intermediate steps for debugging/transparency\n",
        "        print(\"\\n--- Video Monitoring Processing Steps ---\")\n",
        "        for item in orchestrator_result.new_items:\n",
        "            if isinstance(item, MessageOutputItem):\n",
        "                text = ItemHelpers.text_message_output(item)\n",
        "                if text:\n",
        "                    print(f\"  - Processing step: {text}\")\n",
        "\n",
        "        print(\n",
        "            f\"\\n\\n--- Final Video Monitoring Response ---\\n{orchestrator_result.final_output}\"\n",
        "        )\n",
        "        print()\n",
        "\n",
        "    return orchestrator_result.final_output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Simple test function\n",
        "async def test_video_assistant():\n",
        "    \"\"\"Simple test function to demonstrate usage\"\"\"\n",
        "    print(\"🧪 Testing Video Monitoring Assistant\")\n",
        "\n",
        "    # Test basic status\n",
        "    response1 = await video_monitoring_assistant(\n",
        "        \"What's the current monitoring status?\"\n",
        "    )\n",
        "    print(f\"Status response: {response1}\")\n",
        "\n",
        "    # Test YouTube info\n",
        "    response2 = await video_monitoring_assistant(\n",
        "        \"Get stream info for this YouTube URL: https://www.youtube.com/watch?v=jcEW98mEqmY\"\n",
        "    )\n",
        "    print(f\"YouTube info response: {response2}\")\n",
        "\n",
        "    return response1, response2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "result = await get_youtube_info_example(\"https://www.youtube.com/watch?v=YDfiTGGPYCk\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "responses = await quick_start_youtube(\"https://www.youtube.com/watch?v=YDfiTGGPYCk\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "display_result = await Runner.run(\n",
        "    video_orchestrator, \"Show the video feed with overlays\"\n",
        ")\n",
        "print(display_result.final_output)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Show current frame from NBC livestream\n",
        "frame_result = await Runner.run(video_orchestrator, \"Show the latest frame\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Get detailed incident analysis and reports\n",
        "analysis = await Runner.run(video_orchestrator, \"Analyze detected incidents\")\n",
        "print(analysis.final_output)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Get complete monitoring summary\n",
        "summary = await Runner.run(video_orchestrator, \"Create a video summary\")\n",
        "print(summary.final_output)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "result = await Runner.run(video_orchestrator, \"Show recent incidents\")\n",
        "result.final_output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "result = await Runner.run(\n",
        "    video_orchestrator, \"What's the current incident detection status?\"\n",
        ")\n",
        "result.final_output"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.5"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "state": {}
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}
